[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:16:30 dist-test-slave-s12s krb5kdc[20897](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:16:30 dist-test-slave-s12s krb5kdc[20897](info): set up 2 sockets
Sep 25 01:16:30 dist-test-slave-s12s krb5kdc[20897](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 01:16:31.137034 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:16:31 dist-test-slave-s12s krb5kdc[20897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:16:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:16:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:16:31.314810 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:35259
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:35259
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:31.640215 20910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:31.640503 20910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:31.640664 20910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:31.647965 20910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:16:31.648070 20910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:31.648167 20910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:31.648272 20910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:16:31.648357 20910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:16:31.658466 20910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:35259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:35259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:31.659891 20910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:31.661303 20910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:31.673223 20915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:31.673185 20918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:31.673257 20916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:31.674203 20910 server_base.cc:1034] running on GCE node
I20260925 01:16:31.674916 20910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:31.676335 20910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:31.677544 20910 hybrid_clock.cc:648] HybridClock initialized: now 1790298991677470 us; error 55 us; skew 500 ppm
I20260925 01:16:31.677897 20910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:31 dist-test-slave-s12s krb5kdc[20897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298991, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:31.681143 20910 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:16:31.682634 20910 webserver.cc:460] Webserver started at http://127.20.102.190:35233/ using document root <none> and password file <none>
I20260925 01:16:31.683279 20910 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:31.683377 20910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:31.683627 20910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:31.686616 20910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "500629b9f4bb48d1b72877bd63382eb0"
format_stamp: "Formatted at 2026-09-25 01:16:31 on dist-test-slave-s12s"
server_key: "5e4a09bdf89ec1b155356a90055b2adc"
server_key_iv: "ecf055ae88e52d93d87506a765cba27d"
server_key_version: "encryptionkey@0"
I20260925 01:16:31.687315 20910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "500629b9f4bb48d1b72877bd63382eb0"
format_stamp: "Formatted at 2026-09-25 01:16:31 on dist-test-slave-s12s"
server_key: "5e4a09bdf89ec1b155356a90055b2adc"
server_key_iv: "ecf055ae88e52d93d87506a765cba27d"
server_key_version: "encryptionkey@0"
I20260925 01:16:31.692378 20910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 01:16:31.695801 20925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:31.697237 20910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 01:16:31.697422 20910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "500629b9f4bb48d1b72877bd63382eb0"
format_stamp: "Formatted at 2026-09-25 01:16:31 on dist-test-slave-s12s"
server_key: "5e4a09bdf89ec1b155356a90055b2adc"
server_key_iv: "ecf055ae88e52d93d87506a765cba27d"
server_key_version: "encryptionkey@0"
I20260925 01:16:31.697623 20910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:31.735917 20910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:31.740090 20910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:31.740406 20910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:31.759593 20910 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:35259
I20260925 01:16:31.759625 20976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:35259 every 8 connection(s)
I20260925 01:16:31.761073 20910 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:16:31.762194 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 20910
I20260925 01:16:31.762563 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:16:31.764428 20890 external_mini_cluster.cc:1428] Setting key 74602397d2b4eb9b7f1f40ba2f7100f6
I20260925 01:16:31.765206 20977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:31.775972 20977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0: Bootstrap starting.
I20260925 01:16:31.779708 20977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:31.781073 20977 log.cc:824] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0: Log is configured to *not* fsync() on all Append() calls
I20260925 01:16:31.783982 20977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0: No bootstrap required, opened a new log
I20260925 01:16:31.792469 20977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500629b9f4bb48d1b72877bd63382eb0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35259 } }
I20260925 01:16:31.792923 20977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:31.793030 20977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 500629b9f4bb48d1b72877bd63382eb0, State: Initialized, Role: FOLLOWER
Sep 25 01:16:31 dist-test-slave-s12s krb5kdc[20897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:31.793740 20977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500629b9f4bb48d1b72877bd63382eb0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35259 } }
I20260925 01:16:31.794018 20977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:16:31.794170 20977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:16:31.794353 20977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:31.797857 20977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500629b9f4bb48d1b72877bd63382eb0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35259 } }
I20260925 01:16:31.798509 20977 leader_election.cc:302] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 500629b9f4bb48d1b72877bd63382eb0; no voters:
I20260925 01:16:31.799199 20977 leader_election.cc:288] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:16:31.799324 20982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:31.800827 20982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [term 1 LEADER]: Becoming Leader. State: Replica: 500629b9f4bb48d1b72877bd63382eb0, State: Running, Role: LEADER
I20260925 01:16:31.801599 20977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:16:31.801499 20982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500629b9f4bb48d1b72877bd63382eb0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35259 } }
I20260925 01:16:31.807085 20984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 500629b9f4bb48d1b72877bd63382eb0. Latest consensus state: current_term: 1 leader_uuid: "500629b9f4bb48d1b72877bd63382eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500629b9f4bb48d1b72877bd63382eb0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35259 } } }
I20260925 01:16:31.806911 20983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "500629b9f4bb48d1b72877bd63382eb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "500629b9f4bb48d1b72877bd63382eb0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35259 } } }
I20260925 01:16:31.807431 20983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:16:31.809592 20984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:16:31.813603 20979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:31.768374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38604 (local address 127.20.102.190:35259)
0925 01:16:31.769275 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:31.769297 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:31.770651 (+ 1354us) server_negotiation.cc:407] Connection header received
0925 01:16:31.780132 (+ 9481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:31.780166 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:31.780707 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:31.781175 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:31.784413 (+ 3238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:31.786510 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:31.788162 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:31.789235 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:31.789419 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:31.795210 (+ 5791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:31.795221 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:31.795255 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:31.795308 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:31.799422 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:31.800528 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:31.800536 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:31.800543 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:31.800788 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:31.801400 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:31.801405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:31.801408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:31.802280 (+ 872us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:31.802599 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:31.806028 (+ 3429us) server_negotiation.cc:299] Negotiation successful
0925 01:16:31.806446 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":398,"threads_started":1}
I20260925 01:16:31.813894 20995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:16:31.819482 20995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:16:31.828336 20995 catalog_manager.cc:1347] Generated new cluster ID: ed877b69014a4d84a29b015de3a4b2c8
I20260925 01:16:31.828464 20995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:16:31.872818 20995 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:16:31.874225 20995 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:16:31.885092 20995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 500629b9f4bb48d1b72877bd63382eb0: Generated new TSK 0
I20260925 01:16:31.886001 20995 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:16:31.940364 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35259
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:32.283629 21003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:32.283890 21003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:32.283986 21003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:32.291147 21003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:32.291255 21003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:32.291384 21003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:16:32.301404 21003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:32.302815 21003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:32.304163 21003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:32.317787 21011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:32.317845 21008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:32.317879 21009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:32.318573 21003 server_base.cc:1034] running on GCE node
I20260925 01:16:32.319218 21003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:32.320108 21003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:32.321349 21003 hybrid_clock.cc:648] HybridClock initialized: now 1790298992321300 us; error 59 us; skew 500 ppm
I20260925 01:16:32.321734 21003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:32 dist-test-slave-s12s krb5kdc[20897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298992, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:32.324867 21003 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:16:32.326799 21003 webserver.cc:460] Webserver started at http://127.20.102.129:41737/ using document root <none> and password file <none>
I20260925 01:16:32.327559 21003 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:32.327662 21003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:32.327912 21003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:32.330793 21003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "4f017bd2e9394b0ca1fe065c0c9dfab2"
format_stamp: "Formatted at 2026-09-25 01:16:32 on dist-test-slave-s12s"
server_key: "69fe3d3e95dfb504021690009a3a1c91"
server_key_iv: "e75054af5d28583de7b3ca4a3ebf2e17"
server_key_version: "encryptionkey@0"
I20260925 01:16:32.331506 21003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "4f017bd2e9394b0ca1fe065c0c9dfab2"
format_stamp: "Formatted at 2026-09-25 01:16:32 on dist-test-slave-s12s"
server_key: "69fe3d3e95dfb504021690009a3a1c91"
server_key_iv: "e75054af5d28583de7b3ca4a3ebf2e17"
server_key_version: "encryptionkey@0"
I20260925 01:16:32.336493 21003 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:16:32.339805 21018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:32.341240 21003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:16:32.341413 21003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "4f017bd2e9394b0ca1fe065c0c9dfab2"
format_stamp: "Formatted at 2026-09-25 01:16:32 on dist-test-slave-s12s"
server_key: "69fe3d3e95dfb504021690009a3a1c91"
server_key_iv: "e75054af5d28583de7b3ca4a3ebf2e17"
server_key_version: "encryptionkey@0"
I20260925 01:16:32.341598 21003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:32.361688 21003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:32.366003 21003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:32.366315 21003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:32.367339 21003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:16:32.368768 21003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:32.368847 21003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:32.369033 21003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:32.369115 21003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:32.404935 21003 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:41137
I20260925 01:16:32.405033 21130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:41137 every 8 connection(s)
I20260925 01:16:32.406312 21003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:16:32.413793 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 21003
I20260925 01:16:32.414042 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:16:32.414762 20890 external_mini_cluster.cc:1428] Setting key 43d41714bff59f2e283cba2ab01036bb
Sep 25 01:16:32 dist-test-slave-s12s krb5kdc[20897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298992, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:32.427645 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:32.409108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:39531 (local address 127.20.102.190:35259)
0925 01:16:32.409608 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:32.409612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:32.412537 (+ 2925us) server_negotiation.cc:407] Connection header received
0925 01:16:32.412600 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:32.412604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:32.412739 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:32.412833 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:32.414491 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:32.415575 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:32.417618 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:32.418308 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:32.418466 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:32.421896 (+ 3430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:32.421901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:32.421904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:32.421935 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:32.424502 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:32.425249 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:32.425256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:32.425259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:32.425378 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:32.425818 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:32.425823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:32.425826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:32.426171 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:32.426340 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:32.427057 (+ 717us) server_negotiation.cc:299] Negotiation successful
0925 01:16:32.427277 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":268,"threads_started":1}
I20260925 01:16:32.428580 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:32.409538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35259 (local address 127.20.102.129:39531)
0925 01:16:32.410354 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:32.410413 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:32.412259 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:32.413069 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:32.413077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:32.413634 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:32.414300 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:32.414318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:32.415748 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:32.415753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:32.417467 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:32.417480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:32.418519 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:32.418524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:32.418722 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:32.419396 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:32.419424 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:32.421670 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:32.424653 (+ 2983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:32.424662 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:32.424700 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:32.425129 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:32.425492 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:32.425497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:32.425501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:32.425697 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:32.426304 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:32.426312 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:32.426661 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 01:16:32.426943 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 01:16:32.427472 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":261,"threads_started":1}
I20260925 01:16:32.430183 21131 heartbeater.cc:347] Connected to a master server at 127.20.102.190:35259
I20260925 01:16:32.430517 21131 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
I20260925 01:16:32.431329 21131 heartbeater.cc:511] Master 127.20.102.190:35259 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
I20260925 01:16:32.433619 20942 ts_manager.cc:194] Registered new tserver with Master: 4f017bd2e9394b0ca1fe065c0c9dfab2 (127.20.102.129:41137)
I20260925 01:16:32.435005 20942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:39531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:16:32.455245 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35259
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:32.782936 21137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:32.783217 21137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:32.783403 21137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:32.790863 21137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:32.791002 21137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:32.791190 21137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:16:32.801358 21137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:32.802762 21137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:32.804171 21137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:32.817408 21143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:32.817520 21145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:32.817520 21142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:32.818506 21137 server_base.cc:1034] running on GCE node
I20260925 01:16:32.819180 21137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:32.820127 21137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:32.821378 21137 hybrid_clock.cc:648] HybridClock initialized: now 1790298992821336 us; error 60 us; skew 500 ppm
I20260925 01:16:32.821729 21137 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:32 dist-test-slave-s12s krb5kdc[20897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298992, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:32.824877 21137 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:16:32.826313 21137 webserver.cc:460] Webserver started at http://127.20.102.130:44417/ using document root <none> and password file <none>
I20260925 01:16:32.827121 21137 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:32.827265 21137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:32.827598 21137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:32.830529 21137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "b296a35a89054d7189e6f18caa66af9e"
format_stamp: "Formatted at 2026-09-25 01:16:32 on dist-test-slave-s12s"
server_key: "7b90650a033a647eb417e2f1ce223b7c"
server_key_iv: "a85839307e5d6f34fb08b9fbae6949b1"
server_key_version: "encryptionkey@0"
I20260925 01:16:32.831287 21137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "b296a35a89054d7189e6f18caa66af9e"
format_stamp: "Formatted at 2026-09-25 01:16:32 on dist-test-slave-s12s"
server_key: "7b90650a033a647eb417e2f1ce223b7c"
server_key_iv: "a85839307e5d6f34fb08b9fbae6949b1"
server_key_version: "encryptionkey@0"
I20260925 01:16:32.836242 21137 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:16:32.839656 21152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:32.841048 21137 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:16:32.841218 21137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "b296a35a89054d7189e6f18caa66af9e"
format_stamp: "Formatted at 2026-09-25 01:16:32 on dist-test-slave-s12s"
server_key: "7b90650a033a647eb417e2f1ce223b7c"
server_key_iv: "a85839307e5d6f34fb08b9fbae6949b1"
server_key_version: "encryptionkey@0"
I20260925 01:16:32.841416 21137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:32.869820 21137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:32.874089 21137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:32.874395 21137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:32.875397 21137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:16:32.876890 21137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:32.877000 21137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:32.877156 21137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:32.877238 21137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:32.912488 21137 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:41661
I20260925 01:16:32.912555 21264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:41661 every 8 connection(s)
I20260925 01:16:32.913851 21137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:16:32.921022 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 21137
I20260925 01:16:32.921226 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:16:32.921661 20890 external_mini_cluster.cc:1428] Setting key 51ba4f2029104e549e3dc8dbe4081156
Sep 25 01:16:32 dist-test-slave-s12s krb5kdc[20897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298992, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:32.935026 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:32.916431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:34853 (local address 127.20.102.190:35259)
0925 01:16:32.916624 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:32.916629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:32.917869 (+ 1240us) server_negotiation.cc:407] Connection header received
0925 01:16:32.919726 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:32.919731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:32.919825 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:32.919935 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:32.921625 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:32.922488 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:32.924517 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:32.925345 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:32.925517 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:32.929202 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:32.929208 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:32.929211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:32.929241 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:32.931877 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:32.932620 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:32.932631 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:32.932634 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:32.932830 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:32.933276 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:32.933281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:32.933285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:32.933560 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:32.933714 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:32.934419 (+ 705us) server_negotiation.cc:299] Negotiation successful
0925 01:16:32.934678 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 01:16:32.936038 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:32.916992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35259 (local address 127.20.102.130:34853)
0925 01:16:32.917712 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:32.917749 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:32.919448 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:32.920180 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:32.920188 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:32.920769 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:32.921450 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:32.921476 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:32.922675 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:32.922680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:32.924335 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:32.924347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:32.925562 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:32.925567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:32.925760 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:32.926432 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:32.926474 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:32.928986 (+ 2512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:32.932049 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:32.932058 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:32.932075 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:32.932494 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:32.932946 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:32.932951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:32.932955 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:32.933158 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:32.933690 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:32.933697 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:32.934065 (+ 368us) client_negotiation.cc:769] Sending connection context
0925 01:16:32.934326 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 01:16:32.934883 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":274,"threads_started":1}
I20260925 01:16:32.937886 21265 heartbeater.cc:347] Connected to a master server at 127.20.102.190:35259
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
I20260925 01:16:32.938230 21265 heartbeater.cc:464] Registering TS with master...
I20260925 01:16:32.939043 21265 heartbeater.cc:511] Master 127.20.102.190:35259 requested a full tablet report, sending...
I20260925 01:16:32.940554 20942 ts_manager.cc:194] Registered new tserver with Master: b296a35a89054d7189e6f18caa66af9e (127.20.102.130:41661)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
I20260925 01:16:32.941511 20942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:34853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:16:32.967973 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35259
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:35161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:33.297796 21270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:33.298074 21270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:33.298241 21270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:33.305897 21270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:33.306005 21270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:33.306203 21270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:16:33.316120 21270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:35161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35259
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:33.317481 21270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:33.318819 21270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:33.331453 21276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:33.331514 21278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:33.332028 21275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:33.332424 21270 server_base.cc:1034] running on GCE node
I20260925 01:16:33.333032 21270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:33.334002 21270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:33.335242 21270 hybrid_clock.cc:648] HybridClock initialized: now 1790298993335193 us; error 38 us; skew 500 ppm
I20260925 01:16:33.335587 21270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:33 dist-test-slave-s12s krb5kdc[20897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298993, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:33.338718 21270 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:16:33.340101 21270 webserver.cc:460] Webserver started at http://127.20.102.131:38805/ using document root <none> and password file <none>
I20260925 01:16:33.340837 21270 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:33.340979 21270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:33.341284 21270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:33.344216 21270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "ec5225c9a2044c5ab7b1cc0f24cdddea"
format_stamp: "Formatted at 2026-09-25 01:16:33 on dist-test-slave-s12s"
server_key: "0d9a2f1c5e6df61af5c70ed1bb3bdc9a"
server_key_iv: "7855c94aeb0003e8b191d805ec73f379"
server_key_version: "encryptionkey@0"
I20260925 01:16:33.345216 21270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "ec5225c9a2044c5ab7b1cc0f24cdddea"
format_stamp: "Formatted at 2026-09-25 01:16:33 on dist-test-slave-s12s"
server_key: "0d9a2f1c5e6df61af5c70ed1bb3bdc9a"
server_key_iv: "7855c94aeb0003e8b191d805ec73f379"
server_key_version: "encryptionkey@0"
I20260925 01:16:33.350319 21270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:16:33.353732 21285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:33.354984 21270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:16:33.355151 21270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "ec5225c9a2044c5ab7b1cc0f24cdddea"
format_stamp: "Formatted at 2026-09-25 01:16:33 on dist-test-slave-s12s"
server_key: "0d9a2f1c5e6df61af5c70ed1bb3bdc9a"
server_key_iv: "7855c94aeb0003e8b191d805ec73f379"
server_key_version: "encryptionkey@0"
I20260925 01:16:33.355370 21270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:33.375486 21270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:33.380100 21270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:33.380412 21270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:33.381434 21270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:16:33.383105 21270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:33.383227 21270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:33.383361 21270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:33.383422 21270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:33.418260 21270 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:37419
I20260925 01:16:33.418347 21397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:37419 every 8 connection(s)
I20260925 01:16:33.419576 21270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:16:33.425073 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 21270
I20260925 01:16:33.425305 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:16:33.425786 20890 external_mini_cluster.cc:1428] Setting key 27b005367447dc30dfed24fb9111f6b0
Sep 25 01:16:33 dist-test-slave-s12s krb5kdc[20897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298993, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:33.438424 21131 heartbeater.cc:503] Master 127.20.102.190:35259 was elected leader, sending a full tablet report...
I20260925 01:16:33.439528 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:33.422143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:36475 (local address 127.20.102.190:35259)
0925 01:16:33.422339 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:33.422347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:33.423368 (+ 1021us) server_negotiation.cc:407] Connection header received
0925 01:16:33.425304 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:33.425310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:33.425407 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:33.425539 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:33.427359 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:33.428563 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:33.430412 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:33.430819 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:33.430915 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:33.434127 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:33.434131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:33.434134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:33.434165 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:33.436832 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:33.437453 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:33.437456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:33.437459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:33.437573 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:33.437918 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:33.437921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:33.437923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:33.438164 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:33.438267 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:33.439117 (+ 850us) server_negotiation.cc:299] Negotiation successful
0925 01:16:33.439286 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:16:33.440665 21400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:33.422497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35259 (local address 127.20.102.131:36475)
0925 01:16:33.423198 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:33.423242 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:33.425032 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:33.425776 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:33.425784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:33.426337 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:33.427077 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:33.427111 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:33.428765 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:33.428771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:33.430277 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:33.430291 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:33.431018 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:33.431023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:33.431210 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:33.431823 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:33.431850 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:33.433916 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:33.436954 (+ 3038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:33.436960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:33.436971 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:33.437353 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:33.437661 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:33.437664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:33.437666 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:33.437830 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:33.438290 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:33.438296 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:33.438724 (+ 428us) client_negotiation.cc:769] Sending connection context
0925 01:16:33.438977 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 01:16:33.439464 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":272,"threads_started":1}
I20260925 01:16:33.442175 21398 heartbeater.cc:347] Connected to a master server at 127.20.102.190:35259
I20260925 01:16:33.442519 21398 heartbeater.cc:464] Registering TS with master...
I20260925 01:16:33.443219 21398 heartbeater.cc:511] Master 127.20.102.190:35259 requested a full tablet report, sending...
I20260925 01:16:33.444636 20942 ts_manager.cc:194] Registered new tserver with Master: ec5225c9a2044c5ab7b1cc0f24cdddea (127.20.102.131:37419)
I20260925 01:16:33.445623 20942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:36475
I20260925 01:16:33.454744 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:16:33 dist-test-slave-s12s krb5kdc[20897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298993, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 01:16:33 dist-test-slave-s12s krb5kdc[20897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298993, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:16:33.501665 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:33.485176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49630 (local address 127.20.102.129:41137)
0925 01:16:33.485875 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:33.485883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:33.485918 (+ 35us) server_negotiation.cc:407] Connection header received
0925 01:16:33.486003 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:33.486012 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:33.486236 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:33.486408 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:33.487864 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:33.488963 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:33.490217 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:33.490810 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:33.490956 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:33.495481 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:33.495494 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:33.495510 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:33.495548 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:33.498552 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:33.499269 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:33.499284 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:33.499286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:33.499427 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:33.499862 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:33.499868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:33.499870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:33.500235 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:33.500584 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:33.500947 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 01:16:33.501264 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":359,"threads_started":1}
W20260925 01:16:33.503168 21046 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49630
Sep 25 01:16:33 dist-test-slave-s12s krb5kdc[20897](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 01:16:33 dist-test-slave-s12s krb5kdc[20897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:16:33.547770 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:33.533811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49632 (local address 127.20.102.129:41137)
0925 01:16:33.534048 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:33.534057 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:33.534393 (+ 336us) server_negotiation.cc:407] Connection header received
0925 01:16:33.534614 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:33.534618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:33.534713 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:33.534833 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:33.536218 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:33.537013 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:33.538236 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:33.538660 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:33.538796 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:33.542740 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:33.542745 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:33.542749 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:33.542782 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:33.545480 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:33.546144 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:33.546148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:33.546150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:33.546270 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:33.546593 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:33.546596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:33.546598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:33.546828 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:33.546997 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:33.547265 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 01:16:33.547425 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:16:33.549530 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 21003
I20260925 01:16:33.562592 21405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:33.550533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49638 (local address 127.20.102.129:41137)
0925 01:16:33.550703 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:33.550707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:33.550973 (+ 266us) server_negotiation.cc:407] Connection header received
0925 01:16:33.551090 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:33.551094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:33.551152 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:33.551241 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:33.552356 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:33.553195 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:33.554404 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:33.554866 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:33.554979 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:33.557227 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:33.557232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:33.557235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:33.557265 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:33.560050 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:33.560705 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:33.560710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:33.560715 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:33.560851 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:33.561213 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:33.561221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:33.561226 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:33.561479 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:33.561734 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:33.562023 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 01:16:33.562240 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:16:33.563299 21126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:16:33.648566 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 21003
I20260925 01:16:33.666949 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 21137
Sep 25 01:16:33 dist-test-slave-s12s krb5kdc[20897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:16:33.683619 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:33.667861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40696 (local address 127.20.102.130:41661)
0925 01:16:33.668449 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:33.668456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:33.668482 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:16:33.668559 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:33.668565 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:33.668834 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:33.668965 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:33.670292 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:33.671408 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:33.672814 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:33.673370 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:33.673469 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:33.677743 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:33.677755 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:33.677769 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:33.677802 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:33.680768 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:33.681440 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:33.681455 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:33.681457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:33.681589 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:33.681998 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:33.682005 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:33.682008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:33.682343 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:33.682678 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:33.682995 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 01:16:33.683260 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":320,"threads_started":1}
I20260925 01:16:33.685004 21260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:16:33.768294 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 21137
I20260925 01:16:33.791497 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 21270
Sep 25 01:16:33 dist-test-slave-s12s krb5kdc[20897](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:16:33.808832 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:33.792760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55368 (local address 127.20.102.131:37419)
0925 01:16:33.793489 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:33.793496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:33.793530 (+ 34us) server_negotiation.cc:407] Connection header received
0925 01:16:33.793598 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:33.793605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:33.793828 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:33.793976 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:33.795203 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:33.796345 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:33.797658 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:33.798324 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:33.798432 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:33.802851 (+ 4419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:33.802863 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:33.802879 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:33.802919 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:33.806270 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:33.806916 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:33.806923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:33.806926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:33.807066 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:33.807397 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:33.807400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:33.807401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:33.807748 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:33.807973 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:33.808250 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 01:16:33.808455 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":349,"threads_started":1}
I20260925 01:16:33.810168 21393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:16:33.892491 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 21270
I20260925 01:16:33.909682 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 20910
I20260925 01:16:33.910962 20972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:16:34.014048 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 20910
2026-09-25T01:16:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3983 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:16:34 dist-test-slave-s12s krb5kdc[21419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:16:34 dist-test-slave-s12s krb5kdc[21419](info): set up 2 sockets
Sep 25 01:16:34 dist-test-slave-s12s krb5kdc[21419](info): commencing operation
krb5kdc: starting...
W20260925 01:16:35.116822 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:16:35 dist-test-slave-s12s krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:16:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:16:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:16:35.302003 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:38757
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:38757
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:35.629055 21432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:35.629359 21432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:35.629536 21432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:35.637328 21432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:16:35.637506 21432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:35.637605 21432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:35.637708 21432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:16:35.637774 21432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:16:35.648034 21432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:38757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:38757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:35.649549 21432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:35.650908 21432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:35.662806 21437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:35.662887 21438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:35.662851 21440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:35.664126 21432 server_base.cc:1034] running on GCE node
I20260925 01:16:35.665017 21432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:35.666585 21432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:35.667848 21432 hybrid_clock.cc:648] HybridClock initialized: now 1790298995667800 us; error 54 us; skew 500 ppm
I20260925 01:16:35.668252 21432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:35 dist-test-slave-s12s krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298995, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:35.677376 21432 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:16:35.679152 21432 webserver.cc:460] Webserver started at http://127.20.102.190:40577/ using document root <none> and password file <none>
I20260925 01:16:35.679915 21432 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:35.680071 21432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:35.680404 21432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:35.683640 21432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "80f3ad85d863415181175423fb39cc24"
format_stamp: "Formatted at 2026-09-25 01:16:35 on dist-test-slave-s12s"
server_key: "cd6e86c319ce1f5e7c7ff4ae9ac082fb"
server_key_iv: "a87466f4c404da86de18bd2304cec2a4"
server_key_version: "encryptionkey@0"
I20260925 01:16:35.684437 21432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "80f3ad85d863415181175423fb39cc24"
format_stamp: "Formatted at 2026-09-25 01:16:35 on dist-test-slave-s12s"
server_key: "cd6e86c319ce1f5e7c7ff4ae9ac082fb"
server_key_iv: "a87466f4c404da86de18bd2304cec2a4"
server_key_version: "encryptionkey@0"
I20260925 01:16:35.689821 21432 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 01:16:35.693379 21447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:35.694710 21432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:16:35.694900 21432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "80f3ad85d863415181175423fb39cc24"
format_stamp: "Formatted at 2026-09-25 01:16:35 on dist-test-slave-s12s"
server_key: "cd6e86c319ce1f5e7c7ff4ae9ac082fb"
server_key_iv: "a87466f4c404da86de18bd2304cec2a4"
server_key_version: "encryptionkey@0"
I20260925 01:16:35.695097 21432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:35.708873 21432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:35.713094 21432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:35.713378 21432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:35.731866 21432 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:38757
I20260925 01:16:35.731884 21498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:38757 every 8 connection(s)
I20260925 01:16:35.733412 21432 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:16:35.737345 21499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:35.739313 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 21432
I20260925 01:16:35.739543 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:16:35.740191 20890 external_mini_cluster.cc:1428] Setting key e744ace933e435745655de84b0eaa8d1
I20260925 01:16:35.749498 21499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24: Bootstrap starting.
I20260925 01:16:35.753523 21499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:35.754835 21499 log.cc:824] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:16:35 dist-test-slave-s12s krb5kdc[21419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:35.758626 21499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24: No bootstrap required, opened a new log
I20260925 01:16:35.765254 21502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:35.742219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35244 (local address 127.20.102.190:38757)
0925 01:16:35.743026 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:35.743041 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:35.743082 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:16:35.744814 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:35.744838 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:35.745215 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:35.745557 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:35.747406 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:35.748776 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:35.750554 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:35.751227 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:35.751364 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:35.756436 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:35.756448 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:35.756468 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:35.756553 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:35.760449 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:35.761206 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:35.761216 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:35.761227 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:35.761445 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:35.761798 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:35.761809 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:35.761815 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:35.762545 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:35.762844 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:35.763269 (+ 425us) server_negotiation.cc:299] Negotiation successful
0925 01:16:35.763733 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":394,"threads_started":1}
I20260925 01:16:35.769213 21499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f3ad85d863415181175423fb39cc24" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 38757 } }
I20260925 01:16:35.769564 21499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:35.769645 21499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80f3ad85d863415181175423fb39cc24, State: Initialized, Role: FOLLOWER
I20260925 01:16:35.770184 21499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f3ad85d863415181175423fb39cc24" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 38757 } }
I20260925 01:16:35.770390 21499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:16:35.770478 21499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:16:35.770606 21499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:35.773268 21499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f3ad85d863415181175423fb39cc24" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 38757 } }
I20260925 01:16:35.773720 21499 leader_election.cc:302] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80f3ad85d863415181175423fb39cc24; no voters:
I20260925 01:16:35.774235 21499 leader_election.cc:288] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:16:35.774401 21504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:35.775065 21504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [term 1 LEADER]: Becoming Leader. State: Replica: 80f3ad85d863415181175423fb39cc24, State: Running, Role: LEADER
I20260925 01:16:35.775622 21504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f3ad85d863415181175423fb39cc24" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 38757 } }
I20260925 01:16:35.776317 21499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:16:35.778784 21506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80f3ad85d863415181175423fb39cc24. Latest consensus state: current_term: 1 leader_uuid: "80f3ad85d863415181175423fb39cc24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f3ad85d863415181175423fb39cc24" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 38757 } } }
I20260925 01:16:35.778784 21505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80f3ad85d863415181175423fb39cc24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80f3ad85d863415181175423fb39cc24" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 38757 } } }
I20260925 01:16:35.779237 21506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [sys.catalog]: This master's current role is: LEADER
I20260925 01:16:35.779259 21505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24 [sys.catalog]: This master's current role is: LEADER
I20260925 01:16:35.781430 21510 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:16:35.787603 21510 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:16:35.796555 21510 catalog_manager.cc:1347] Generated new cluster ID: 2950f12bd0244bf39296c83438b52024
I20260925 01:16:35.796669 21510 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:16:35.827555 21510 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:16:35.829442 21510 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:16:35.837496 21510 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80f3ad85d863415181175423fb39cc24: Generated new TSK 0
I20260925 01:16:35.838413 21510 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:16:35.890918 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:38757
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:36.237004 21525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:36.237370 21525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:36.237555 21525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:36.244740 21525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:36.244901 21525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:36.245096 21525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:16:36.254788 21525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:38757
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:36.256220 21525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:36.257751 21525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:36.271323 21531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:36.271386 21530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:36.271565 21533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:36.272447 21525 server_base.cc:1034] running on GCE node
I20260925 01:16:36.273241 21525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:36.274217 21525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:36.275450 21525 hybrid_clock.cc:648] HybridClock initialized: now 1790298996275368 us; error 64 us; skew 500 ppm
I20260925 01:16:36.275806 21525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:36 dist-test-slave-s12s krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:36.279927 21525 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:16:36.281757 21525 webserver.cc:460] Webserver started at http://127.20.102.129:45621/ using document root <none> and password file <none>
I20260925 01:16:36.282508 21525 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:36.282663 21525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:36.283042 21525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:36.286789 21525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "a9711154a85e4e1baf91f2a1d3062cd7"
format_stamp: "Formatted at 2026-09-25 01:16:36 on dist-test-slave-s12s"
server_key: "a02200c9b5a135b89b1673bda435a315"
server_key_iv: "3de38a49667cac32d0d4347126a460df"
server_key_version: "encryptionkey@0"
I20260925 01:16:36.287709 21525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "a9711154a85e4e1baf91f2a1d3062cd7"
format_stamp: "Formatted at 2026-09-25 01:16:36 on dist-test-slave-s12s"
server_key: "a02200c9b5a135b89b1673bda435a315"
server_key_iv: "3de38a49667cac32d0d4347126a460df"
server_key_version: "encryptionkey@0"
I20260925 01:16:36.293619 21525 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 01:16:36.297180 21540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:36.298509 21525 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:16:36.298707 21525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "a9711154a85e4e1baf91f2a1d3062cd7"
format_stamp: "Formatted at 2026-09-25 01:16:36 on dist-test-slave-s12s"
server_key: "a02200c9b5a135b89b1673bda435a315"
server_key_iv: "3de38a49667cac32d0d4347126a460df"
server_key_version: "encryptionkey@0"
I20260925 01:16:36.298904 21525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:36.315112 21525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:36.319694 21525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:36.319988 21525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:36.321012 21525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:16:36.323473 21525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:36.323553 21525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:36.323690 21525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:36.323745 21525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:36.355726 21525 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:45335
I20260925 01:16:36.355762 21652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:45335 every 8 connection(s)
I20260925 01:16:36.357137 21525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:16:36.362628 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 21525
I20260925 01:16:36.362875 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:16:36.363612 20890 external_mini_cluster.cc:1428] Setting key 8a082ae39f8b1f92b13c59978e1f893f
Sep 25 01:16:36 dist-test-slave-s12s krb5kdc[21419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:36.379433 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:36.359812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:52355 (local address 127.20.102.190:38757)
0925 01:16:36.360212 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:36.360216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:36.361291 (+ 1075us) server_negotiation.cc:407] Connection header received
0925 01:16:36.363187 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:36.363192 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:36.363284 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:36.363425 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:36.365175 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:36.366089 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:36.367620 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:36.367995 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:36.368125 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:36.372049 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:36.372059 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:36.372068 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:36.372115 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:36.375736 (+ 3621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:36.376521 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:36.376529 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:36.376536 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:36.376710 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:36.377213 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:36.377221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:36.377227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:36.377628 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:36.377791 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:36.378637 (+ 846us) server_negotiation.cc:299] Negotiation successful
0925 01:16:36.378935 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":191,"threads_started":1}
I20260925 01:16:36.380225 21656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:36.360365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:38757 (local address 127.20.102.129:52355)
0925 01:16:36.361142 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:36.361180 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:36.362859 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:36.363705 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:36.363718 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:36.364306 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:36.365008 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:36.365026 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:36.366256 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:36.366261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:36.367473 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:36.367484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:36.368217 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:36.368225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:36.368465 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:36.369194 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:36.369218 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:36.371819 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:36.375887 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:36.375894 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:36.375909 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:36.376373 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:36.376824 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:36.376828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:36.376830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:36.377079 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:36.377836 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:36.377843 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:36.378189 (+ 346us) client_negotiation.cc:769] Sending connection context
0925 01:16:36.378446 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 01:16:36.378893 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":352,"threads_started":1}
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
I20260925 01:16:36.382042 21653 heartbeater.cc:347] Connected to a master server at 127.20.102.190:38757
I20260925 01:16:36.382347 21653 heartbeater.cc:464] Registering TS with master...
I20260925 01:16:36.383198 21653 heartbeater.cc:511] Master 127.20.102.190:38757 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
I20260925 01:16:36.385828 21464 ts_manager.cc:194] Registered new tserver with Master: a9711154a85e4e1baf91f2a1d3062cd7 (127.20.102.129:45335)
I20260925 01:16:36.388046 21464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:52355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:16:36.407553 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:38757
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:36.741394 21659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:36.741717 21659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:36.741883 21659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:36.749200 21659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:36.749331 21659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:36.749511 21659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:16:36.759343 21659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:38757
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:36.760751 21659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:36.762095 21659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:36.776381 21667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:36.776341 21664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:36.776414 21665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:36.777731 21659 server_base.cc:1034] running on GCE node
I20260925 01:16:36.778415 21659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:36.779393 21659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:36.780623 21659 hybrid_clock.cc:648] HybridClock initialized: now 1790298996780562 us; error 55 us; skew 500 ppm
I20260925 01:16:36.780982 21659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:36 dist-test-slave-s12s krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:36.784215 21659 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:16:36.785796 21659 webserver.cc:460] Webserver started at http://127.20.102.130:41259/ using document root <none> and password file <none>
I20260925 01:16:36.786504 21659 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:36.786654 21659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:36.786969 21659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:36.789968 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "52513735f66c4a69a687e97ea3f543c4"
format_stamp: "Formatted at 2026-09-25 01:16:36 on dist-test-slave-s12s"
server_key: "b64d16c40159fffccfc08ea23341507f"
server_key_iv: "040af218ca2cd6cac635d041547a58c6"
server_key_version: "encryptionkey@0"
I20260925 01:16:36.790750 21659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "52513735f66c4a69a687e97ea3f543c4"
format_stamp: "Formatted at 2026-09-25 01:16:36 on dist-test-slave-s12s"
server_key: "b64d16c40159fffccfc08ea23341507f"
server_key_iv: "040af218ca2cd6cac635d041547a58c6"
server_key_version: "encryptionkey@0"
I20260925 01:16:36.795861 21659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:16:36.799682 21674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:36.801407 21659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:16:36.801653 21659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "52513735f66c4a69a687e97ea3f543c4"
format_stamp: "Formatted at 2026-09-25 01:16:36 on dist-test-slave-s12s"
server_key: "b64d16c40159fffccfc08ea23341507f"
server_key_iv: "040af218ca2cd6cac635d041547a58c6"
server_key_version: "encryptionkey@0"
I20260925 01:16:36.801880 21659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:36.830660 21659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:36.835155 21659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:36.835469 21659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:36.836576 21659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:16:36.838105 21659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:36.838205 21659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:36.838348 21659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:36.838420 21659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:36.871701 21659 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:42657
I20260925 01:16:36.871748 21786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:42657 every 8 connection(s)
I20260925 01:16:36.873062 21659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:16:36.876380 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 21659
I20260925 01:16:36.876586 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:16:36.877261 20890 external_mini_cluster.cc:1428] Setting key 9c673cee2b73d5d6e5eaa488196b7a55
Sep 25 01:16:36 dist-test-slave-s12s krb5kdc[21419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298996, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:36.894017 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:36.875659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:33149 (local address 127.20.102.190:38757)
0925 01:16:36.875839 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:36.875844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:36.877176 (+ 1332us) server_negotiation.cc:407] Connection header received
0925 01:16:36.879299 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:36.879304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:36.879423 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:36.879540 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:36.881156 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:36.881950 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:36.883600 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:36.884039 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:36.884166 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:36.888019 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:36.888025 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:36.888028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:36.888071 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:36.890790 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:36.891572 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:36.891577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:36.891579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:36.891704 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:36.892157 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:36.892165 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:36.892168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:36.892454 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:36.892618 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:36.893405 (+ 787us) server_negotiation.cc:299] Negotiation successful
0925 01:16:36.893654 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
I20260925 01:16:36.895087 21789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:36.876151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:38757 (local address 127.20.102.130:33149)
0925 01:16:36.877000 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:36.877048 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:36.878968 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:36.879745 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:36.879752 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:36.880263 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:36.880979 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:36.881007 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:36.882099 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:36.882104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:36.883449 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:36.883462 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:36.884263 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:36.884271 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:36.884499 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:36.885213 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:36.885235 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:36.887774 (+ 2539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:36.890952 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:36.890962 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:36.890980 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:36.891428 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:36.891820 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:36.891826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:36.891829 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:36.892036 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:36.892591 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:36.892600 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:36.893042 (+ 442us) client_negotiation.cc:769] Sending connection context
0925 01:16:36.893313 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 01:16:36.893883 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":333,"threads_started":1}
I20260925 01:16:36.896623 21787 heartbeater.cc:347] Connected to a master server at 127.20.102.190:38757
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
I20260925 01:16:36.897056 21787 heartbeater.cc:464] Registering TS with master...
I20260925 01:16:36.898042 21787 heartbeater.cc:511] Master 127.20.102.190:38757 requested a full tablet report, sending...
I20260925 01:16:36.899822 21464 ts_manager.cc:194] Registered new tserver with Master: 52513735f66c4a69a687e97ea3f543c4 (127.20.102.130:42657)
I20260925 01:16:36.901088 21464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:33149
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:16:36.919785 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:38757
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:37.250077 21792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:37.250370 21792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:37.250466 21792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:37.257684 21792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:37.257788 21792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:37.257898 21792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:16:37.267679 21792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:38757
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:37.268970 21792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:37.270267 21792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:37.284106 21798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:37.284147 21800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:37.284171 21797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:37.285248 21792 server_base.cc:1034] running on GCE node
I20260925 01:16:37.285807 21792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:37.286736 21792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:37.287931 21792 hybrid_clock.cc:648] HybridClock initialized: now 1790298997287867 us; error 56 us; skew 500 ppm
I20260925 01:16:37.288267 21792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:37 dist-test-slave-s12s krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:37.291447 21792 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:16:37.293043 21792 webserver.cc:460] Webserver started at http://127.20.102.131:35269/ using document root <none> and password file <none>
I20260925 01:16:37.293711 21792 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:37.293855 21792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:37.294169 21792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:37.297163 21792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "772b3b72a31a4e3294aa25553275ce01"
format_stamp: "Formatted at 2026-09-25 01:16:37 on dist-test-slave-s12s"
server_key: "de02f9acde92068f3604e812080e1d68"
server_key_iv: "f73454cf71d54d79de52e820cd0d301a"
server_key_version: "encryptionkey@0"
I20260925 01:16:37.297962 21792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "772b3b72a31a4e3294aa25553275ce01"
format_stamp: "Formatted at 2026-09-25 01:16:37 on dist-test-slave-s12s"
server_key: "de02f9acde92068f3604e812080e1d68"
server_key_iv: "f73454cf71d54d79de52e820cd0d301a"
server_key_version: "encryptionkey@0"
I20260925 01:16:37.303486 21792 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:16:37.307354 21807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:37.308740 21792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 01:16:37.308908 21792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "772b3b72a31a4e3294aa25553275ce01"
format_stamp: "Formatted at 2026-09-25 01:16:37 on dist-test-slave-s12s"
server_key: "de02f9acde92068f3604e812080e1d68"
server_key_iv: "f73454cf71d54d79de52e820cd0d301a"
server_key_version: "encryptionkey@0"
I20260925 01:16:37.309131 21792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:37.342417 21792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:37.346661 21792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:37.346967 21792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:37.347968 21792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:16:37.349380 21792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:37.349486 21792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:37.349686 21792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:37.349761 21792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:37.384663 21792 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:41745
I20260925 01:16:37.384799 21919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:41745 every 8 connection(s)
I20260925 01:16:37.386039 21792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:16:37.387506 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 21792
I20260925 01:16:37.387665 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:16:37.388053 20890 external_mini_cluster.cc:1428] Setting key f428d386f4b82ca51c2ec23822243742
I20260925 01:16:37.391239 21653 heartbeater.cc:503] Master 127.20.102.190:38757 was elected leader, sending a full tablet report...
Sep 25 01:16:37 dist-test-slave-s12s krb5kdc[21419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298997, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:37.408499 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.390236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:49429 (local address 127.20.102.190:38757)
0925 01:16:37.390415 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:37.390421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:37.392040 (+ 1619us) server_negotiation.cc:407] Connection header received
0925 01:16:37.393991 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:37.393994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:37.394057 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:37.394137 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:37.395730 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.396498 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.398063 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.398590 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.398685 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.402392 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:37.402398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:37.402401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:37.402434 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:37.405545 (+ 3111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.406286 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.406290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.406292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.406413 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.406928 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.406932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.406934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.407211 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:37.407320 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:37.408042 (+ 722us) server_negotiation.cc:299] Negotiation successful
0925 01:16:37.408207 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 01:16:37.409512 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.390745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:38757 (local address 127.20.102.131:49429)
0925 01:16:37.391877 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:37.391924 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:37.393696 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:37.394397 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:37.394406 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:37.394931 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:37.395571 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:37.395589 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.396630 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.396634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:37.397926 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:37.397938 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.398811 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.398817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:37.399029 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.399700 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:37.399727 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:37.402112 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:37.405672 (+ 3560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:37.405679 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:37.405693 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:37.406178 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:37.406543 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:37.406547 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:37.406550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:37.406766 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:37.407336 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:37.407343 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:37.407686 (+ 343us) client_negotiation.cc:769] Sending connection context
0925 01:16:37.407926 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 01:16:37.408327 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"thread_start_us":684,"threads_started":1}
I20260925 01:16:37.410987 21920 heartbeater.cc:347] Connected to a master server at 127.20.102.190:38757
I20260925 01:16:37.411302 21920 heartbeater.cc:464] Registering TS with master...
I20260925 01:16:37.412071 21920 heartbeater.cc:511] Master 127.20.102.190:38757 requested a full tablet report, sending...
I20260925 01:16:37.413672 21464 ts_manager.cc:194] Registered new tserver with Master: 772b3b72a31a4e3294aa25553275ce01 (127.20.102.131:41745)
I20260925 01:16:37.414778 21464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:49429
I20260925 01:16:37.416360 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:16:37.435014 21655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.422524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.20.102.190:38757)
0925 01:16:37.422679 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:37.422683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:37.423147 (+ 464us) server_negotiation.cc:407] Connection header received
0925 01:16:37.423271 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:37.423275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:37.423336 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:37.423418 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:37.424839 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.425556 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.426805 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.427256 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.427366 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.429718 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:37.429724 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:37.429727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:37.429759 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:37.432451 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.433175 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.433179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.433181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.433303 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.433682 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.433685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.433687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.433994 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:37.434179 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:37.434499 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 01:16:37.434651 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:16:37.441588 21464 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:16:37.448427 21464 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:16:37.482666 21932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.461437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:41745 (local address 127.0.0.1:56924)
0925 01:16:37.470162 (+ 8725us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:37.470217 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:37.470402 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:37.474742 (+ 4340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:37.474747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:37.474788 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:37.475285 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:37.475299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.477228 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.477234 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:37.479781 (+ 2547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:37.479797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.481637 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.481647 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:37.481860 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.481893 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:16:37.481962 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 01:16:37.482126 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8464,"mutex_wait_us":33,"thread_start_us":8299,"threads_started":1}
I20260925 01:16:37.482645 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.462640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.20.102.129:45335)
0925 01:16:37.473346 (+ 10706us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:37.473360 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:37.474603 (+ 1243us) server_negotiation.cc:407] Connection header received
0925 01:16:37.474818 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:37.474829 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:37.475000 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:37.475214 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:37.475869 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.477494 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.480266 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.481519 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.481805 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.482007 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:37.482119 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 01:16:37.482282 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10442,"thread_start_us":11161,"threads_started":1}
I20260925 01:16:37.482724 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.462852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.20.102.130:42657)
0925 01:16:37.472105 (+ 9253us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:37.472119 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:37.472160 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:16:37.472296 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:37.472311 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:37.473881 (+ 1570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:37.474719 (+ 838us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:37.475005 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.476725 (+ 1720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.479581 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.481059 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.481230 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.481498 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:37.482097 (+ 599us) server_negotiation.cc:299] Negotiation successful
0925 01:16:37.482336 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9033,"thread_start_us":8851,"threads_started":1}
I20260925 01:16:37.482688 21936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.462605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:45335 (local address 127.0.0.1:35340)
0925 01:16:37.474460 (+ 11855us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:37.474482 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:37.474614 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:37.475201 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:37.475205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:37.475251 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:37.475680 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:37.475695 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.477776 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.477782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:37.480091 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:37.480102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.481808 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.481814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:37.481968 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.481994 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 01:16:37.482086 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 01:16:37.482179 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11708,"mutex_wait_us":7,"thread_start_us":11546,"threads_started":1}
I20260925 01:16:37.483566 21934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.462341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:42657 (local address 127.0.0.1:56044)
0925 01:16:37.471208 (+ 8867us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:37.471240 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:37.471385 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:37.474158 (+ 2773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:37.474169 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:37.474212 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:37.474702 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:37.474715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.476861 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.476866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:37.479415 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:37.479437 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.481319 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.481327 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:37.481547 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.481670 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 01:16:37.481835 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 01:16:37.483027 (+ 1192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8692,"mutex_wait_us":206,"thread_start_us":8494,"threads_started":1}
I20260925 01:16:37.484274 21933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.461766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.20.102.131:41745)
0925 01:16:37.474026 (+ 12260us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:37.474039 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:37.474086 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:16:37.474202 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:37.474211 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:37.474489 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:37.475539 (+ 1050us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:37.475821 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.477077 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.479956 (+ 2879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.481379 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.483546 (+ 2167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.483721 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:37.483785 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 01:16:37.483932 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12036,"thread_start_us":11826,"threads_started":1}
I20260925 01:16:37.487347 21722 tablet_service.cc:1467] Processing CreateTablet for tablet 35485751ef45419a9df4617f58a84ffd (DEFAULT_TABLE table=test-table [id=f5cd56af6c8248108bd2b2fa4bf5e76f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:37.487589 21855 tablet_service.cc:1467] Processing CreateTablet for tablet 35485751ef45419a9df4617f58a84ffd (DEFAULT_TABLE table=test-table [id=f5cd56af6c8248108bd2b2fa4bf5e76f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:37.487581 21588 tablet_service.cc:1467] Processing CreateTablet for tablet 35485751ef45419a9df4617f58a84ffd (DEFAULT_TABLE table=test-table [id=f5cd56af6c8248108bd2b2fa4bf5e76f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:37.488751 21722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35485751ef45419a9df4617f58a84ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:37.488974 21588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35485751ef45419a9df4617f58a84ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:37.488989 21855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35485751ef45419a9df4617f58a84ffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:37.500365 21938 tablet_bootstrap.cc:492] T 35485751ef45419a9df4617f58a84ffd P 52513735f66c4a69a687e97ea3f543c4: Bootstrap starting.
I20260925 01:16:37.503515 21939 tablet_bootstrap.cc:492] T 35485751ef45419a9df4617f58a84ffd P a9711154a85e4e1baf91f2a1d3062cd7: Bootstrap starting.
I20260925 01:16:37.505514 21938 tablet_bootstrap.cc:654] T 35485751ef45419a9df4617f58a84ffd P 52513735f66c4a69a687e97ea3f543c4: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:37.506884 21938 log.cc:824] T 35485751ef45419a9df4617f58a84ffd P 52513735f66c4a69a687e97ea3f543c4: Log is configured to *not* fsync() on all Append() calls
I20260925 01:16:37.508165 21939 tablet_bootstrap.cc:654] T 35485751ef45419a9df4617f58a84ffd P a9711154a85e4e1baf91f2a1d3062cd7: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:37.508409 21940 tablet_bootstrap.cc:492] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01: Bootstrap starting.
I20260925 01:16:37.509478 21939 log.cc:824] T 35485751ef45419a9df4617f58a84ffd P a9711154a85e4e1baf91f2a1d3062cd7: Log is configured to *not* fsync() on all Append() calls
I20260925 01:16:37.510353 21938 tablet_bootstrap.cc:492] T 35485751ef45419a9df4617f58a84ffd P 52513735f66c4a69a687e97ea3f543c4: No bootstrap required, opened a new log
I20260925 01:16:37.510646 21938 ts_tablet_manager.cc:1397] T 35485751ef45419a9df4617f58a84ffd P 52513735f66c4a69a687e97ea3f543c4: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260925 01:16:37.512115 21940 tablet_bootstrap.cc:654] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:37.512795 21939 tablet_bootstrap.cc:492] T 35485751ef45419a9df4617f58a84ffd P a9711154a85e4e1baf91f2a1d3062cd7: No bootstrap required, opened a new log
I20260925 01:16:37.513098 21939 ts_tablet_manager.cc:1397] T 35485751ef45419a9df4617f58a84ffd P a9711154a85e4e1baf91f2a1d3062cd7: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 01:16:37.513545 21940 log.cc:824] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01: Log is configured to *not* fsync() on all Append() calls
I20260925 01:16:37.516722 21940 tablet_bootstrap.cc:492] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01: No bootstrap required, opened a new log
I20260925 01:16:37.517004 21940 ts_tablet_manager.cc:1397] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260925 01:16:37.519483 21938 raft_consensus.cc:348] T 35485751ef45419a9df4617f58a84ffd P 52513735f66c4a69a687e97ea3f543c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772b3b72a31a4e3294aa25553275ce01" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 41745 } } peers { permanent_uuid: "52513735f66c4a69a687e97ea3f543c4" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42657 } } peers { permanent_uuid: "a9711154a85e4e1baf91f2a1d3062cd7" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45335 } }
I20260925 01:16:37.519853 21938 raft_consensus.cc:374] T 35485751ef45419a9df4617f58a84ffd P 52513735f66c4a69a687e97ea3f543c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:37.519942 21938 raft_consensus.cc:729] T 35485751ef45419a9df4617f58a84ffd P 52513735f66c4a69a687e97ea3f543c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52513735f66c4a69a687e97ea3f543c4, State: Initialized, Role: FOLLOWER
I20260925 01:16:37.520542 21938 consensus_queue.cc:260] T 35485751ef45419a9df4617f58a84ffd P 52513735f66c4a69a687e97ea3f543c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772b3b72a31a4e3294aa25553275ce01" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 41745 } } peers { permanent_uuid: "52513735f66c4a69a687e97ea3f543c4" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42657 } } peers { permanent_uuid: "a9711154a85e4e1baf91f2a1d3062cd7" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45335 } }
I20260925 01:16:37.521708 21787 heartbeater.cc:503] Master 127.20.102.190:38757 was elected leader, sending a full tablet report...
I20260925 01:16:37.522256 21938 ts_tablet_manager.cc:1428] T 35485751ef45419a9df4617f58a84ffd P 52513735f66c4a69a687e97ea3f543c4: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260925 01:16:37.525478 21939 raft_consensus.cc:348] T 35485751ef45419a9df4617f58a84ffd P a9711154a85e4e1baf91f2a1d3062cd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772b3b72a31a4e3294aa25553275ce01" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 41745 } } peers { permanent_uuid: "52513735f66c4a69a687e97ea3f543c4" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42657 } } peers { permanent_uuid: "a9711154a85e4e1baf91f2a1d3062cd7" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45335 } }
I20260925 01:16:37.525935 21939 raft_consensus.cc:374] T 35485751ef45419a9df4617f58a84ffd P a9711154a85e4e1baf91f2a1d3062cd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:37.526052 21939 raft_consensus.cc:729] T 35485751ef45419a9df4617f58a84ffd P a9711154a85e4e1baf91f2a1d3062cd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9711154a85e4e1baf91f2a1d3062cd7, State: Initialized, Role: FOLLOWER
I20260925 01:16:37.527427 21940 raft_consensus.cc:348] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772b3b72a31a4e3294aa25553275ce01" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 41745 } } peers { permanent_uuid: "52513735f66c4a69a687e97ea3f543c4" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42657 } } peers { permanent_uuid: "a9711154a85e4e1baf91f2a1d3062cd7" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45335 } }
I20260925 01:16:37.527558 21939 consensus_queue.cc:260] T 35485751ef45419a9df4617f58a84ffd P a9711154a85e4e1baf91f2a1d3062cd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772b3b72a31a4e3294aa25553275ce01" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 41745 } } peers { permanent_uuid: "52513735f66c4a69a687e97ea3f543c4" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42657 } } peers { permanent_uuid: "a9711154a85e4e1baf91f2a1d3062cd7" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45335 } }
I20260925 01:16:37.527864 21940 raft_consensus.cc:374] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:37.527971 21940 raft_consensus.cc:729] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 772b3b72a31a4e3294aa25553275ce01, State: Initialized, Role: FOLLOWER
I20260925 01:16:37.529662 21939 ts_tablet_manager.cc:1428] T 35485751ef45419a9df4617f58a84ffd P a9711154a85e4e1baf91f2a1d3062cd7: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
I20260925 01:16:37.533202 21940 consensus_queue.cc:260] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772b3b72a31a4e3294aa25553275ce01" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 41745 } } peers { permanent_uuid: "52513735f66c4a69a687e97ea3f543c4" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42657 } } peers { permanent_uuid: "a9711154a85e4e1baf91f2a1d3062cd7" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45335 } }
I20260925 01:16:37.533519 21946 raft_consensus.cc:482] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:16:37.533784 21946 raft_consensus.cc:504] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772b3b72a31a4e3294aa25553275ce01" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 41745 } } peers { permanent_uuid: "52513735f66c4a69a687e97ea3f543c4" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42657 } } peers { permanent_uuid: "a9711154a85e4e1baf91f2a1d3062cd7" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45335 } }
I20260925 01:16:37.535507 21946 leader_election.cc:288] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52513735f66c4a69a687e97ea3f543c4 (127.20.102.130:42657), a9711154a85e4e1baf91f2a1d3062cd7 (127.20.102.129:45335)
I20260925 01:16:37.536046 21920 heartbeater.cc:503] Master 127.20.102.190:38757 was elected leader, sending a full tablet report...
I20260925 01:16:37.537415 21940 ts_tablet_manager.cc:1428] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01: Time spent starting tablet: real 0.020s user 0.016s sys 0.002s
I20260925 01:16:37.543571 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.536425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:33229 (local address 127.20.102.129:45335)
0925 01:16:37.537377 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:37.537385 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:37.537409 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:16:37.537477 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:37.537485 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:37.537558 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:37.537655 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:37.538157 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.539117 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.541882 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.542665 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.542835 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.542914 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:37.543180 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 01:16:37.543318 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":809}
I20260925 01:16:37.543578 21922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.535778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:45335 (local address 127.20.102.131:33229)
0925 01:16:37.535993 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:37.536014 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:37.536165 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:37.537728 (+ 1563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:37.537733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:37.537764 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:37.537995 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:37.538007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.539275 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.539280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:37.541549 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:37.541564 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.542817 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.542822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:37.543025 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.543054 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 01:16:37.543128 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 01:16:37.543270 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":82}
I20260925 01:16:37.543603 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.536444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:35353 (local address 127.20.102.130:42657)
0925 01:16:37.536630 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:37.536636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:37.537319 (+ 683us) server_negotiation.cc:407] Connection header received
0925 01:16:37.537592 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:37.537598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:37.537669 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:37.537733 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:37.538599 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.539555 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.541916 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.542672 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.542837 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.542913 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:37.543242 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 01:16:37.543369 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 01:16:37.543887 21948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.535916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:42657 (local address 127.20.102.131:35353)
0925 01:16:37.537204 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:37.537241 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:37.537424 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:37.537941 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:37.537945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:37.537968 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:37.538415 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:37.538432 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.539711 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.539717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:37.541777 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:37.541786 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.542895 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.542901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:37.543064 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.543088 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:16:37.543203 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 01:16:37.543330 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"mutex_wait_us":1,"thread_start_us":1017,"threads_started":1}
I20260925 01:16:37.545032 21742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35485751ef45419a9df4617f58a84ffd" candidate_uuid: "772b3b72a31a4e3294aa25553275ce01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52513735f66c4a69a687e97ea3f543c4" is_pre_election: true
I20260925 01:16:37.545022 21608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35485751ef45419a9df4617f58a84ffd" candidate_uuid: "772b3b72a31a4e3294aa25553275ce01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9711154a85e4e1baf91f2a1d3062cd7" is_pre_election: true
I20260925 01:16:37.545610 21742 raft_consensus.cc:2454] T 35485751ef45419a9df4617f58a84ffd P 52513735f66c4a69a687e97ea3f543c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 772b3b72a31a4e3294aa25553275ce01 in term 0.
I20260925 01:16:37.545610 21608 raft_consensus.cc:2454] T 35485751ef45419a9df4617f58a84ffd P a9711154a85e4e1baf91f2a1d3062cd7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 772b3b72a31a4e3294aa25553275ce01 in term 0.
I20260925 01:16:37.546342 21808 leader_election.cc:302] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52513735f66c4a69a687e97ea3f543c4, 772b3b72a31a4e3294aa25553275ce01, a9711154a85e4e1baf91f2a1d3062cd7; no voters:
I20260925 01:16:37.546793 21947 raft_consensus.cc:2781] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:16:37.546968 21947 raft_consensus.cc:482] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:16:37.547107 21947 raft_consensus.cc:3037] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:37.549510 21947 raft_consensus.cc:504] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772b3b72a31a4e3294aa25553275ce01" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 41745 } } peers { permanent_uuid: "52513735f66c4a69a687e97ea3f543c4" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42657 } } peers { permanent_uuid: "a9711154a85e4e1baf91f2a1d3062cd7" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45335 } }
I20260925 01:16:37.550118 21947 leader_election.cc:288] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [CANDIDATE]: Term 1 election: Requested vote from peers 52513735f66c4a69a687e97ea3f543c4 (127.20.102.130:42657), a9711154a85e4e1baf91f2a1d3062cd7 (127.20.102.129:45335)
I20260925 01:16:37.550575 21608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35485751ef45419a9df4617f58a84ffd" candidate_uuid: "772b3b72a31a4e3294aa25553275ce01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9711154a85e4e1baf91f2a1d3062cd7"
I20260925 01:16:37.550558 21742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35485751ef45419a9df4617f58a84ffd" candidate_uuid: "772b3b72a31a4e3294aa25553275ce01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52513735f66c4a69a687e97ea3f543c4"
I20260925 01:16:37.550781 21742 raft_consensus.cc:3037] T 35485751ef45419a9df4617f58a84ffd P 52513735f66c4a69a687e97ea3f543c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:37.550783 21608 raft_consensus.cc:3037] T 35485751ef45419a9df4617f58a84ffd P a9711154a85e4e1baf91f2a1d3062cd7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:37.554064 21742 raft_consensus.cc:2454] T 35485751ef45419a9df4617f58a84ffd P 52513735f66c4a69a687e97ea3f543c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 772b3b72a31a4e3294aa25553275ce01 in term 1.
I20260925 01:16:37.554064 21608 raft_consensus.cc:2454] T 35485751ef45419a9df4617f58a84ffd P a9711154a85e4e1baf91f2a1d3062cd7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 772b3b72a31a4e3294aa25553275ce01 in term 1.
I20260925 01:16:37.554505 21809 leader_election.cc:302] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52513735f66c4a69a687e97ea3f543c4, 772b3b72a31a4e3294aa25553275ce01; no voters:
I20260925 01:16:37.554816 21947 raft_consensus.cc:2781] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:37.555044 21947 raft_consensus.cc:686] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [term 1 LEADER]: Becoming Leader. State: Replica: 772b3b72a31a4e3294aa25553275ce01, State: Running, Role: LEADER
I20260925 01:16:37.555552 21947 consensus_queue.cc:237] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772b3b72a31a4e3294aa25553275ce01" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 41745 } } peers { permanent_uuid: "52513735f66c4a69a687e97ea3f543c4" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42657 } } peers { permanent_uuid: "a9711154a85e4e1baf91f2a1d3062cd7" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45335 } }
I20260925 01:16:37.559882 21464 catalog_manager.cc:5495] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 772b3b72a31a4e3294aa25553275ce01 (127.20.102.131). New cstate: current_term: 1 leader_uuid: "772b3b72a31a4e3294aa25553275ce01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "772b3b72a31a4e3294aa25553275ce01" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 41745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52513735f66c4a69a687e97ea3f543c4" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9711154a85e4e1baf91f2a1d3062cd7" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45335 } health_report { overall_health: UNKNOWN } } }
Sep 25 01:16:37 dist-test-slave-s12s krb5kdc[21419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:16:37.603024 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.587896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35350 (local address 127.20.102.129:45335)
0925 01:16:37.588133 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:37.588142 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:37.588618 (+ 476us) server_negotiation.cc:407] Connection header received
0925 01:16:37.588870 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:37.588875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:37.588955 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:37.589055 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:37.590565 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.591336 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.592531 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.593037 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.593133 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.597004 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:37.597020 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:37.597038 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:37.597072 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:37.600194 (+ 3122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.600910 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.600918 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.600920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.601049 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.601437 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.601440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.601443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.601819 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:37.602146 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:37.602441 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 01:16:37.602677 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 01:16:37.609603 21654 tablet.cc:2369] T 35485751ef45419a9df4617f58a84ffd P a9711154a85e4e1baf91f2a1d3062cd7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:16:37.625402 21788 tablet.cc:2369] T 35485751ef45419a9df4617f58a84ffd P 52513735f66c4a69a687e97ea3f543c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 01:16:37 dist-test-slave-s12s krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298997, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260925 01:16:37.637799 21921 tablet.cc:2369] T 35485751ef45419a9df4617f58a84ffd P 772b3b72a31a4e3294aa25553275ce01: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 01:16:37 dist-test-slave-s12s krb5kdc[21419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298997, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:16:37.650957 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.637272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35358 (local address 127.20.102.129:45335)
0925 01:16:37.637524 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:37.637533 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:37.637775 (+ 242us) server_negotiation.cc:407] Connection header received
0925 01:16:37.638115 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:37.638122 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:37.638212 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:37.638292 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:37.639825 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.640568 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.641838 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.642234 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.642330 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.646017 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:37.646022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:37.646025 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:37.646066 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:37.648843 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.649394 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.649397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.649400 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.649519 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.649819 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.649822 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.649823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.650073 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:37.650278 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:37.650441 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 01:16:37.650594 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Sep 25 01:16:37 dist-test-slave-s12s krb5kdc[21419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 01:16:37 dist-test-slave-s12s krb5kdc[21419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:16:37.695695 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.681901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35360 (local address 127.20.102.129:45335)
0925 01:16:37.682157 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:37.682164 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:37.682298 (+ 134us) server_negotiation.cc:407] Connection header received
0925 01:16:37.682447 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:37.682451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:37.682519 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:37.682608 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:37.684083 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.684819 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.686003 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.686429 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.686525 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.690542 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:37.690548 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:37.690551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:37.690588 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:37.693329 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.693970 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.693978 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.693983 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.694138 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.694442 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.694445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.694447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.694728 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:37.694908 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:37.695158 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 01:16:37.695336 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260925 01:16:37.697587 21568 tablet_service.cc:2908] Rejecting scan request for tablet 35485751ef45419a9df4617f58a84ffd: Uninitialized: clean time has not yet been initialized
I20260925 01:16:37.700547 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 21525
I20260925 01:16:37.713400 21935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.701626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35370 (local address 127.20.102.129:45335)
0925 01:16:37.701833 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:37.701842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:37.702224 (+ 382us) server_negotiation.cc:407] Connection header received
0925 01:16:37.702418 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:37.702424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:37.702498 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:37.702596 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:37.703839 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.704529 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.705932 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.706381 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.706521 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.708561 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:37.708566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:37.708569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:37.708597 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:37.711149 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.711729 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.711733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.711735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.711844 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.712217 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.712220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.712222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.712484 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:37.712794 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:37.713016 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 01:16:37.713185 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 01:16:37.714066 21648 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:16:37.803673 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 21525
I20260925 01:16:37.821799 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 21659
Sep 25 01:16:37 dist-test-slave-s12s krb5kdc[21419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:16:37.836575 21937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.822608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56052 (local address 127.20.102.130:42657)
0925 01:16:37.822774 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:37.822779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:37.822797 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:16:37.822954 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:37.822958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:37.823024 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:37.823119 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:37.824325 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.825199 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.826406 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.826783 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.826874 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.830929 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:37.830946 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:37.830964 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:37.831003 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:37.834133 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.834816 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.834824 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.834827 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.834950 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.835332 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.835335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.835337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.835698 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:37.835941 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:37.836161 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 01:16:37.836321 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:16:37.837282 21782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:16:37.926573 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 21659
I20260925 01:16:37.944788 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 21792
Sep 25 01:16:37 dist-test-slave-s12s krb5kdc[21419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790298997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:16:37.959962 21933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:37.945530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56940 (local address 127.20.102.131:41745)
0925 01:16:37.945735 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:37.945744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:37.945765 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:16:37.945997 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:37.946001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:37.946067 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:37.946183 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:37.947438 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.948183 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.949575 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:37.949981 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:37.950120 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:37.954211 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:37.954225 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:37.954242 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:37.954298 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:37.957183 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.957855 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.957862 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.957865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.958026 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:37.958404 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:37.958410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:37.958415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:37.958783 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:37.959093 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:37.959385 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 01:16:37.959627 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:16:37.960754 21915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:16:38.050700 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 21792
I20260925 01:16:38.067830 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 21432
I20260925 01:16:38.069104 21494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:16:38.180200 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 21432
2026-09-25T01:16:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4167 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:16:38 dist-test-slave-s12s krb5kdc[21967](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:16:38 dist-test-slave-s12s krb5kdc[21967](info): set up 2 sockets
Sep 25 01:16:38 dist-test-slave-s12s krb5kdc[21967](info): commencing operation
krb5kdc: starting...
W20260925 01:16:39.329607 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260925 01:16:39.486860 20890 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.157s user 0.001s sys 0.005s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260925 01:16:39.832283 20890 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.246s user 0.000s sys 0.005s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/test-user.keytab.
W20260925 01:16:40.520984 20890 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.688s user 0.004s sys 0.000s
Sep 25 01:16:40 dist-test-slave-s12s krb5kdc[21967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:16:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:16:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
W20260925 01:16:41.572557 20890 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.20.102.190: real 1.003s user 0.004s sys 0.004s
I20260925 01:16:41.574045 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:35757
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:35757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:41.906947 21980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:41.907230 21980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:41.907405 21980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:41.914748 21980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:16:41.914847 21980 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 01:16:41.914891 21980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:41.914951 21980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:41.915000 21980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:16:41.915062 21980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:16:41.925365 21980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:35757
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:35757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:41.926610 21980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:41.927840 21980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:41.939481 21985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:41.939579 21988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:41.939527 21986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:41.940410 21980 server_base.cc:1034] running on GCE node
I20260925 01:16:41.941269 21980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:41.942704 21980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:41.943975 21980 hybrid_clock.cc:648] HybridClock initialized: now 1790299001943902 us; error 56 us; skew 500 ppm
I20260925 01:16:41.944386 21980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:41 dist-test-slave-s12s krb5kdc[21967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299001, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:41.951622 21980 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:16:41.953234 21980 webserver.cc:460] Webserver started at http://127.20.102.190:32915/ using document root <none> and password file <none>
I20260925 01:16:41.953940 21980 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:41.954088 21980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:41.954398 21980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:41.957485 21980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "48dcaa2b79914f8fb5548bc085de626e"
format_stamp: "Formatted at 2026-09-25 01:16:41 on dist-test-slave-s12s"
server_key: "c4b19444bd1618295a90aed055550cf6"
server_key_iv: "9416cfd4a6c8d1d02c1ca3224f0c5692"
server_key_version: "encryptionkey@0"
I20260925 01:16:41.958302 21980 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "48dcaa2b79914f8fb5548bc085de626e"
format_stamp: "Formatted at 2026-09-25 01:16:41 on dist-test-slave-s12s"
server_key: "c4b19444bd1618295a90aed055550cf6"
server_key_iv: "9416cfd4a6c8d1d02c1ca3224f0c5692"
server_key_version: "encryptionkey@0"
I20260925 01:16:41.963622 21980 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:16:41.967104 21995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:41.968400 21980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:16:41.968585 21980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "48dcaa2b79914f8fb5548bc085de626e"
format_stamp: "Formatted at 2026-09-25 01:16:41 on dist-test-slave-s12s"
server_key: "c4b19444bd1618295a90aed055550cf6"
server_key_iv: "9416cfd4a6c8d1d02c1ca3224f0c5692"
server_key_version: "encryptionkey@0"
I20260925 01:16:41.968868 21980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:41.984829 21980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:41.989018 21980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:41.989324 21980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:42.012073 21980 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:35757
I20260925 01:16:42.012117 22056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:35757 every 8 connection(s)
I20260925 01:16:42.013477 21980 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:16:42.017403 22057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:42.022486 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 21980
I20260925 01:16:42.022720 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:16:42.023355 20890 external_mini_cluster.cc:1428] Setting key ee9bbe6e973c320370ba84fa7f7f26dc
I20260925 01:16:42.028918 22057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e: Bootstrap starting.
I20260925 01:16:42.032578 22057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:42.033695 22057 log.cc:824] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e: Log is configured to *not* fsync() on all Append() calls
I20260925 01:16:42.036643 22057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e: No bootstrap required, opened a new log
Sep 25 01:16:42 dist-test-slave-s12s krb5kdc[21967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:42.045883 22057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48dcaa2b79914f8fb5548bc085de626e" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35757 } }
I20260925 01:16:42.046362 22057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:42.046507 22057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48dcaa2b79914f8fb5548bc085de626e, State: Initialized, Role: FOLLOWER
I20260925 01:16:42.047075 22057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48dcaa2b79914f8fb5548bc085de626e" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35757 } }
I20260925 01:16:42.047333 22057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:16:42.047479 22057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:16:42.047598 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:42.025364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.20.102.190:35757)
0925 01:16:42.026256 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:42.026270 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:42.026345 (+ 75us) server_negotiation.cc:407] Connection header received
0925 01:16:42.028660 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:42.028711 (+ 51us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:42.029178 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:42.029586 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:42.030859 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:42.032899 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:42.034167 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:42.035009 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:42.035199 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:42.039678 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:42.039686 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:42.039699 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:42.039736 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:42.043109 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:42.043943 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:42.043948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:42.043954 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:42.044137 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:42.044558 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:42.044561 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:42.044563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:42.045120 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:42.045344 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:42.045817 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 01:16:42.046179 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":363,"threads_started":1}
I20260925 01:16:42.047688 22057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:42.074008 22057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48dcaa2b79914f8fb5548bc085de626e" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35757 } }
I20260925 01:16:42.074734 22057 leader_election.cc:302] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48dcaa2b79914f8fb5548bc085de626e; no voters:
I20260925 01:16:42.075451 22057 leader_election.cc:288] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:16:42.075579 22062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:42.076259 22062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [term 1 LEADER]: Becoming Leader. State: Replica: 48dcaa2b79914f8fb5548bc085de626e, State: Running, Role: LEADER
I20260925 01:16:42.076921 22062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48dcaa2b79914f8fb5548bc085de626e" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35757 } }
I20260925 01:16:42.078822 22057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:16:42.080464 22064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48dcaa2b79914f8fb5548bc085de626e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48dcaa2b79914f8fb5548bc085de626e" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35757 } } }
I20260925 01:16:42.080511 22063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48dcaa2b79914f8fb5548bc085de626e. Latest consensus state: current_term: 1 leader_uuid: "48dcaa2b79914f8fb5548bc085de626e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48dcaa2b79914f8fb5548bc085de626e" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35757 } } }
I20260925 01:16:42.080901 22064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [sys.catalog]: This master's current role is: LEADER
I20260925 01:16:42.081054 22063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e [sys.catalog]: This master's current role is: LEADER
I20260925 01:16:42.083542 22067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:16:42.089118 22067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:16:42.098127 22067 catalog_manager.cc:1347] Generated new cluster ID: 1af09637140240c8b080ffec458d1541
I20260925 01:16:42.098237 22067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:16:42.108732 22067 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:16:42.110064 22067 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:16:42.119709 22067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e: Generated new TSK 0
I20260925 01:16:42.120736 22067 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
W20260925 01:16:43.314247 20890 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.20.102.129: real 1.189s user 0.000s sys 0.010s
I20260925 01:16:43.315315 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35757
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:43.640141 22083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:43.640412 22083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:43.640522 22083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:43.647923 22083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:43.648039 22083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:43.648121 22083 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:16:43.648202 22083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:16:43.658295 22083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.102.190:35757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:43.659569 22083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:43.660941 22083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:43.674317 22088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:43.674228 22091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:43.674263 22089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:43.675359 22083 server_base.cc:1034] running on GCE node
I20260925 01:16:43.676011 22083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:43.676931 22083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:43.678194 22083 hybrid_clock.cc:648] HybridClock initialized: now 1790299003678121 us; error 30 us; skew 500 ppm
I20260925 01:16:43.678488 22083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:43 dist-test-slave-s12s krb5kdc[21967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299003, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:43.682147 22083 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:16:43.683753 22083 webserver.cc:460] Webserver started at http://127.20.102.129:39525/ using document root <none> and password file <none>
I20260925 01:16:43.684520 22083 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:43.684643 22083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:43.685024 22083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:43.688140 22083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "ad56dbd458cf4f90a6cc794677996fb4"
format_stamp: "Formatted at 2026-09-25 01:16:43 on dist-test-slave-s12s"
server_key: "8828b108f5be057ca8338a9c4760d376"
server_key_iv: "2def31afd2477617dd866db03a88f6d0"
server_key_version: "encryptionkey@0"
I20260925 01:16:43.689069 22083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "ad56dbd458cf4f90a6cc794677996fb4"
format_stamp: "Formatted at 2026-09-25 01:16:43 on dist-test-slave-s12s"
server_key: "8828b108f5be057ca8338a9c4760d376"
server_key_iv: "2def31afd2477617dd866db03a88f6d0"
server_key_version: "encryptionkey@0"
I20260925 01:16:43.694571 22083 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 01:16:43.698256 22098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:43.699668 22083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:16:43.699859 22083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "ad56dbd458cf4f90a6cc794677996fb4"
format_stamp: "Formatted at 2026-09-25 01:16:43 on dist-test-slave-s12s"
server_key: "8828b108f5be057ca8338a9c4760d376"
server_key_iv: "2def31afd2477617dd866db03a88f6d0"
server_key_version: "encryptionkey@0"
I20260925 01:16:43.700096 22083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:43.721583 22083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:43.726027 22083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:43.726322 22083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:43.729238 22083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:43.729352 22083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:43.729557 22083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:43.729712 22083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 25 01:16:43 dist-test-slave-s12s krb5kdc[21967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299003, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:43.754769 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:43.729682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:58947 (local address 127.20.102.190:35757)
0925 01:16:43.730219 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:43.730228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:43.731470 (+ 1242us) server_negotiation.cc:407] Connection header received
0925 01:16:43.733527 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:43.733541 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:43.733629 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:43.733748 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:43.736023 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:43.737604 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:43.740001 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:43.740533 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:43.740667 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:43.746401 (+ 5734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:43.746412 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:43.746419 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:43.746477 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:43.750770 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:43.751833 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:43.751840 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:43.751843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:43.752021 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:43.752900 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:43.752907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:43.752912 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:43.753239 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:43.753407 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:43.754140 (+ 733us) server_negotiation.cc:299] Negotiation successful
0925 01:16:43.754367 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":265,"threads_started":1}
I20260925 01:16:43.755898 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:43.730468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35757 (local address 127.20.102.129:58947)
0925 01:16:43.731286 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:43.731326 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:43.733253 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:43.733991 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:43.733999 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:43.734746 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:43.735683 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:43.735718 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:43.737760 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:43.737765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:43.739791 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:43.739811 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:43.740905 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:43.740913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:43.741285 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:43.742044 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:43.742083 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:43.745978 (+ 3895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:43.750967 (+ 4989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:43.750978 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:43.751002 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:43.751624 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:43.752152 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:43.752161 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:43.752165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:43.752602 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:43.753387 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:43.753395 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:43.753774 (+ 379us) client_negotiation.cc:769] Sending connection context
0925 01:16:43.754052 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 01:16:43.754626 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":357,"threads_started":1}
I20260925 01:16:43.776154 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:43.762156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:46899 (local address 127.20.102.190:35757)
0925 01:16:43.762406 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:43.762415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:43.762980 (+ 565us) server_negotiation.cc:407] Connection header received
0925 01:16:43.763223 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:43.763227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:43.763287 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:43.763393 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:43.764923 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:43.765618 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:43.766929 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:43.767357 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:43.767484 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:43.769553 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:43.769558 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:43.769562 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:43.769591 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:43.772753 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:43.773747 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:43.773753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:43.773757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:43.773905 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:43.774482 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:43.774490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:43.774586 (+ 96us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:43.774959 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:43.775129 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:43.775567 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 01:16:43.775789 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 01:16:43.776309 22176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:43.762088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35757 (local address 127.20.102.129:46899)
0925 01:16:43.762828 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:43.762882 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:43.763077 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:43.763422 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:43.763427 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:43.764009 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:43.764780 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:43.764792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:43.765761 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:43.765767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:43.766784 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:43.766795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:43.767551 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:43.767559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:43.767714 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:43.768320 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:43.768338 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:43.769287 (+ 949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:43.772966 (+ 3679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:43.772974 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:43.772980 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:43.773591 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:43.774050 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:43.774059 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:43.774067 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:43.774286 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:43.775091 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:43.775096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:43.775307 (+ 211us) client_negotiation.cc:769] Sending connection context
0925 01:16:43.775455 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 01:16:43.775812 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":350,"threads_started":1}
I20260925 01:16:43.787770 22083 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:40283
I20260925 01:16:43.787789 22218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:40283 every 8 connection(s)
I20260925 01:16:43.789163 22083 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:16:43.791420 22219 heartbeater.cc:347] Connected to a master server at 127.20.102.190:35757
I20260925 01:16:43.791719 22219 heartbeater.cc:464] Registering TS with master...
I20260925 01:16:43.792454 22219 heartbeater.cc:511] Master 127.20.102.190:35757 requested a full tablet report, sending...
I20260925 01:16:43.793892 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 22083
I20260925 01:16:43.794121 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:16:43.794698 22012 ts_manager.cc:194] Registered new tserver with Master: ad56dbd458cf4f90a6cc794677996fb4 (127.20.102.129:40283)
I20260925 01:16:43.794785 20890 external_mini_cluster.cc:1428] Setting key a2029b22df942f568219a0b66d4af95c
I20260925 01:16:43.796994 22012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:58947
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:16:43.834117 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35757
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:44.156781 22223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:44.157042 22223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:44.157229 22223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:44.164539 22223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:44.164636 22223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:44.164862 22223 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:16:44.164999 22223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:16:44.174549 22223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.102.190:35757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:44.175798 22223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:44.177130 22223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:44.189648 22228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:44.189723 22229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:44.190369 22231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:44.191208 22223 server_base.cc:1034] running on GCE node
I20260925 01:16:44.191864 22223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:44.192734 22223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:44.193984 22223 hybrid_clock.cc:648] HybridClock initialized: now 1790299004193907 us; error 57 us; skew 500 ppm
I20260925 01:16:44.194258 22223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:44 dist-test-slave-s12s krb5kdc[21967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299004, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:44.197475 22223 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:16:44.198904 22223 webserver.cc:460] Webserver started at http://127.20.102.130:39255/ using document root <none> and password file <none>
I20260925 01:16:44.199548 22223 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:44.199630 22223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:44.199884 22223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:44.202672 22223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "f0d0e2fb66b04a509e58367147779926"
format_stamp: "Formatted at 2026-09-25 01:16:44 on dist-test-slave-s12s"
server_key: "674b75b1f216027a2d28d4cccff0fab7"
server_key_iv: "710474120fd26c6d81102ef4882185b6"
server_key_version: "encryptionkey@0"
I20260925 01:16:44.203370 22223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "f0d0e2fb66b04a509e58367147779926"
format_stamp: "Formatted at 2026-09-25 01:16:44 on dist-test-slave-s12s"
server_key: "674b75b1f216027a2d28d4cccff0fab7"
server_key_iv: "710474120fd26c6d81102ef4882185b6"
server_key_version: "encryptionkey@0"
I20260925 01:16:44.208410 22223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 01:16:44.211714 22238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:44.213063 22223 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 01:16:44.213233 22223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "f0d0e2fb66b04a509e58367147779926"
format_stamp: "Formatted at 2026-09-25 01:16:44 on dist-test-slave-s12s"
server_key: "674b75b1f216027a2d28d4cccff0fab7"
server_key_iv: "710474120fd26c6d81102ef4882185b6"
server_key_version: "encryptionkey@0"
I20260925 01:16:44.213443 22223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:44.230417 22223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:44.234387 22223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:44.234664 22223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:44.237323 22223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:44.237430 22223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:44.237607 22223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:44.237681 22223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:16:44 dist-test-slave-s12s krb5kdc[21967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299004, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:44.262702 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.238271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:35643 (local address 127.20.102.190:35757)
0925 01:16:44.238488 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:44.238494 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:44.239823 (+ 1329us) server_negotiation.cc:407] Connection header received
0925 01:16:44.242336 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:44.242343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:44.242433 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:44.242548 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:44.245502 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.246591 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.249364 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.249802 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.249975 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.254159 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:44.254168 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:44.254175 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:44.254224 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:44.258356 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:44.259465 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:44.259471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:44.259476 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:44.259625 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:44.260329 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:44.260336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:44.260341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:44.260782 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:44.260953 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:44.262059 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0925 01:16:44.262292 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260925 01:16:44.264346 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.238850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35757 (local address 127.20.102.130:35643)
0925 01:16:44.239669 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:44.239706 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:44.241964 (+ 2258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:44.243054 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:44.243070 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:44.244137 (+ 1067us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:44.245197 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:44.245234 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.246856 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.246860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:44.249114 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:44.249131 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.250056 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.250068 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:44.250307 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.251019 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:44.251044 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:44.253822 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:44.258517 (+ 4695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:44.258531 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:44.258547 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:44.259293 (+ 746us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:44.259766 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:44.259771 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:44.259780 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:44.260140 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:44.261042 (+ 902us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:44.261054 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:44.261581 (+ 527us) client_negotiation.cc:769] Sending connection context
0925 01:16:44.261959 (+ 378us) client_negotiation.cc:240] Negotiation successful
0925 01:16:44.262581 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":381,"threads_started":1}
I20260925 01:16:44.286592 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.270855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:46011 (local address 127.20.102.190:35757)
0925 01:16:44.271067 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:44.271075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:44.271780 (+ 705us) server_negotiation.cc:407] Connection header received
0925 01:16:44.272004 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:44.272010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:44.272085 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:44.272195 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:44.273707 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.274738 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.276060 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.276682 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.276838 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.279701 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:44.279706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:44.279710 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:44.279738 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:44.283768 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:44.284658 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:44.284667 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:44.284670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:44.284838 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:44.285197 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:44.285203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:44.285206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:44.285523 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:44.285666 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:44.286046 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 01:16:44.286247 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:16:44.286689 22313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.270893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35757 (local address 127.20.102.130:46011)
0925 01:16:44.271664 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:44.271691 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:44.271855 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:44.272230 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:44.272234 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:44.272960 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:44.273522 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:44.273534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.274878 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.274884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:44.275904 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:44.275915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.277121 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.277134 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:44.277374 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.278097 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:44.278133 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:44.279460 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:44.283919 (+ 4459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:44.283925 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:44.283932 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:44.284497 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:44.284947 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:44.284953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:44.284956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:44.285090 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:44.285647 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:44.285652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:44.285820 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 01:16:44.285957 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 01:16:44.286256 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":429,"threads_started":1}
I20260925 01:16:44.292546 22223 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:36899
I20260925 01:16:44.292573 22357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:36899 every 8 connection(s)
I20260925 01:16:44.293743 22223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:16:44.295603 22358 heartbeater.cc:347] Connected to a master server at 127.20.102.190:35757
I20260925 01:16:44.295830 22358 heartbeater.cc:464] Registering TS with master...
I20260925 01:16:44.296446 22358 heartbeater.cc:511] Master 127.20.102.190:35757 requested a full tablet report, sending...
I20260925 01:16:44.297806 22012 ts_manager.cc:194] Registered new tserver with Master: f0d0e2fb66b04a509e58367147779926 (127.20.102.130:36899)
I20260925 01:16:44.298666 22012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:35643
I20260925 01:16:44.300807 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 22223
I20260925 01:16:44.300953 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:16:44.301342 20890 external_mini_cluster.cc:1428] Setting key 4d615f9bd83c28500702fee6e5dad09d
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:16:44.339952 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35757
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:40695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:44.660161 22362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:44.660422 22362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:44.660593 22362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:44.667650 22362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:44.667762 22362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:44.667910 22362 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:16:44.668041 22362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:16:44.677484 22362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:40695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.102.190:35757
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:44.678725 22362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:44.680065 22362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:44.693133 22367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:44.693195 22368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:44.693233 22370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:44.694420 22362 server_base.cc:1034] running on GCE node
I20260925 01:16:44.695019 22362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:44.695888 22362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:44.697173 22362 hybrid_clock.cc:648] HybridClock initialized: now 1790299004697077 us; error 42 us; skew 500 ppm
I20260925 01:16:44.697784 22362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:44 dist-test-slave-s12s krb5kdc[21967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299004, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:44.702121 22362 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:16:44.704010 22362 webserver.cc:460] Webserver started at http://127.20.102.131:43157/ using document root <none> and password file <none>
I20260925 01:16:44.704969 22362 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:44.705101 22362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:44.705428 22362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:44.709798 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "1d0672a4ca904c2782dbe9b1aa061f28"
format_stamp: "Formatted at 2026-09-25 01:16:44 on dist-test-slave-s12s"
server_key: "9e79ff9430f38dce589d8243207c9c9d"
server_key_iv: "cb0f0598557c1c61e582500e2224d353"
server_key_version: "encryptionkey@0"
I20260925 01:16:44.710850 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "1d0672a4ca904c2782dbe9b1aa061f28"
format_stamp: "Formatted at 2026-09-25 01:16:44 on dist-test-slave-s12s"
server_key: "9e79ff9430f38dce589d8243207c9c9d"
server_key_iv: "cb0f0598557c1c61e582500e2224d353"
server_key_version: "encryptionkey@0"
I20260925 01:16:44.717676 22362 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260925 01:16:44.722198 22377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:44.723944 22362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260925 01:16:44.724116 22362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "1d0672a4ca904c2782dbe9b1aa061f28"
format_stamp: "Formatted at 2026-09-25 01:16:44 on dist-test-slave-s12s"
server_key: "9e79ff9430f38dce589d8243207c9c9d"
server_key_iv: "cb0f0598557c1c61e582500e2224d353"
server_key_version: "encryptionkey@0"
I20260925 01:16:44.724380 22362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:44.740211 22362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:44.745843 22362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:44.746119 22362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:44.748528 22362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:44.748639 22362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:44.748814 22362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:44.748896 22362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:16:44 dist-test-slave-s12s krb5kdc[21967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299004, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:44.772330 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.750542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:46877 (local address 127.20.102.190:35757)
0925 01:16:44.750754 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:44.750761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:44.750784 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:16:44.752887 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:44.752892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:44.752979 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:44.753112 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:44.754578 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.755904 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.759652 (+ 3748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.760123 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.760224 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.765312 (+ 5088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:44.765317 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:44.765320 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:44.765353 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:44.769109 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:44.769860 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:44.769866 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:44.769869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:44.770010 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:44.770475 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:44.770481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:44.770484 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:44.770782 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:44.770942 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:44.771739 (+ 797us) server_negotiation.cc:299] Negotiation successful
0925 01:16:44.771971 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 01:16:44.773758 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.749799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35757 (local address 127.20.102.131:46877)
0925 01:16:44.750636 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:44.750680 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:44.752543 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:44.753313 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:44.753320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:44.753775 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:44.754405 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:44.754423 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.756121 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.756127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:44.759469 (+ 3342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:44.759493 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.760373 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.760383 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:44.760782 (+ 399us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.761697 (+ 915us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:44.761736 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:44.765081 (+ 3345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:44.769271 (+ 4190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:44.769283 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:44.769300 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:44.769728 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:44.770124 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:44.770129 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:44.770133 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:44.770351 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:44.770916 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:44.770924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:44.771327 (+ 403us) client_negotiation.cc:769] Sending connection context
0925 01:16:44.771603 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 01:16:44.772163 (+ 560us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":364,"threads_started":1}
I20260925 01:16:44.797739 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.780201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:44179 (local address 127.20.102.190:35757)
0925 01:16:44.780409 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:44.780418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:44.781153 (+ 735us) server_negotiation.cc:407] Connection header received
0925 01:16:44.781479 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:44.781486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:44.781595 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:44.781676 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:44.783396 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.784252 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.786804 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.787392 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.787523 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.790214 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:44.790220 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:44.790226 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:44.790261 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:44.794233 (+ 3972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:44.795266 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:44.795279 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:44.795281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:44.795394 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:44.795809 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:44.795812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:44.795814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:44.796140 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:44.796305 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:44.797171 (+ 866us) server_negotiation.cc:299] Negotiation successful
0925 01:16:44.797424 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:16:44.797854 22448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.780227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35757 (local address 127.20.102.131:44179)
0925 01:16:44.781025 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:44.781048 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:44.781293 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:44.781932 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:44.781943 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:44.782707 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:44.783219 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:44.783238 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.784398 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.784406 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:44.786647 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:44.786668 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.787615 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.787624 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:44.787876 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.788655 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:44.788706 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:44.789993 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:44.794413 (+ 4420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:44.794420 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:44.794424 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:44.795097 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:44.795506 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:44.795512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:44.795517 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:44.795695 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:44.796266 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:44.796271 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:44.796565 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 01:16:44.796769 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 01:16:44.797131 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":395,"threads_started":1}
I20260925 01:16:44.800647 22219 heartbeater.cc:503] Master 127.20.102.190:35757 was elected leader, sending a full tablet report...
I20260925 01:16:44.809222 22362 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:33397
I20260925 01:16:44.809243 22496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:33397 every 8 connection(s)
I20260925 01:16:44.810410 22362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:16:44.812454 22497 heartbeater.cc:347] Connected to a master server at 127.20.102.190:35757
I20260925 01:16:44.812714 22497 heartbeater.cc:464] Registering TS with master...
I20260925 01:16:44.813361 22497 heartbeater.cc:511] Master 127.20.102.190:35757 requested a full tablet report, sending...
I20260925 01:16:44.814606 22012 ts_manager.cc:194] Registered new tserver with Master: 1d0672a4ca904c2782dbe9b1aa061f28 (127.20.102.131:33397)
I20260925 01:16:44.815460 22012 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:46877
I20260925 01:16:44.816614 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 22362
I20260925 01:16:44.816835 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:16:44.817412 20890 external_mini_cluster.cc:1428] Setting key b453d5be1ad9a7e472b7a8690a56b6b7
I20260925 01:16:44.820108 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:16:44 dist-test-slave-s12s krb5kdc[21967](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 01:16:44 dist-test-slave-s12s krb5kdc[21967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:44.864493 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.851385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45796 (local address 127.20.102.190:35757)
0925 01:16:44.851561 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:44.851565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:44.851872 (+ 307us) server_negotiation.cc:407] Connection header received
0925 01:16:44.852022 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:44.852026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:44.852091 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:44.852185 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:44.853684 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.854487 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.855692 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.856086 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.856180 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.859952 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:44.859956 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:44.859959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:44.859989 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:44.862370 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:44.862934 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:44.862938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:44.862940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:44.863049 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:44.863369 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:44.863372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:44.863374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:44.863643 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:44.863821 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:44.864054 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 01:16:44.864219 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:16:44.869019 22012 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:16:44.875933 22012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:16:44.910522 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.889243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.20.102.129:40283)
0925 01:16:44.901543 (+ 12300us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:44.901555 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:44.901593 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:16:44.901721 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:44.901732 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:44.902067 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:44.902330 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:44.903132 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.904919 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.907816 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.909361 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.909543 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.909722 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:44.910052 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 01:16:44.910224 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12083,"thread_start_us":11916,"threads_started":1}
I20260925 01:16:44.910601 22514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.888844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:40283 (local address 127.0.0.1:51406)
0925 01:16:44.900320 (+ 11476us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:44.900335 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:44.900423 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:44.902380 (+ 1957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:44.902391 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:44.902440 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:44.902902 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:44.902914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.905119 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.905126 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:44.907615 (+ 2489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:44.907629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.909582 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.909589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:44.909800 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.909852 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 01:16:44.909988 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 01:16:44.910188 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11373,"mutex_wait_us":2,"thread_start_us":11541,"threads_started":1}
I20260925 01:16:44.911903 22510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.888633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:33397 (local address 127.0.0.1:55504)
0925 01:16:44.900169 (+ 11536us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:44.900191 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:44.900312 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:44.903951 (+ 3639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:44.903959 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:44.903996 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:44.904491 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:44.904502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.906608 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.906616 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:44.908832 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:44.908844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.910726 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.910732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:44.911096 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.911271 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 01:16:44.911355 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 01:16:44.911482 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11374,"mutex_wait_us":269,"thread_start_us":9400,"threads_started":1}
I20260925 01:16:44.912634 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.888894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55504 (local address 127.20.102.131:33397)
0925 01:16:44.903232 (+ 14338us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:44.903244 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:44.903285 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:16:44.903396 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:44.903407 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:44.903697 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:44.904612 (+ 915us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:44.905003 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.906473 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.908977 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.910233 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.910407 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.910567 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:44.912130 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0925 01:16:44.912310 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14106,"thread_start_us":13875,"threads_started":1}
I20260925 01:16:44.912670 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.888200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:36899 (local address 127.0.0.1:41262)
0925 01:16:44.898580 (+ 10380us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:44.898647 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:44.898881 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:44.905193 (+ 6312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:44.905198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:44.905224 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:44.905664 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:44.905676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.907541 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.907552 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:44.909745 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:44.909763 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.911770 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.911778 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:44.912150 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.912182 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:16:44.912257 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 01:16:44.912374 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9988,"thread_start_us":11620,"threads_started":1}
I20260925 01:16:44.913395 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:44.888429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41262 (local address 127.20.102.130:36899)
0925 01:16:44.904577 (+ 16148us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:44.904585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:44.904612 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:16:44.904723 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:44.904730 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:44.904982 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:44.905190 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:44.905869 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.907363 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.909903 (+ 2540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:44.911242 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:44.911422 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:44.911616 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:44.912915 (+ 1299us) server_negotiation.cc:299] Negotiation successful
0925 01:16:44.913087 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15993,"thread_start_us":15874,"threads_started":1}
I20260925 01:16:44.916221 22149 tablet_service.cc:1467] Processing CreateTablet for tablet 59571d3897524aa091298aeaa9c13c63 (DEFAULT_TABLE table=test-table [id=19273c9b883b4c4ca20ea52b5c37f5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:44.917037 22288 tablet_service.cc:1467] Processing CreateTablet for tablet 59571d3897524aa091298aeaa9c13c63 (DEFAULT_TABLE table=test-table [id=19273c9b883b4c4ca20ea52b5c37f5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:44.917824 22149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59571d3897524aa091298aeaa9c13c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:44.918406 22288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59571d3897524aa091298aeaa9c13c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:44.918437 22427 tablet_service.cc:1467] Processing CreateTablet for tablet 59571d3897524aa091298aeaa9c13c63 (DEFAULT_TABLE table=test-table [id=19273c9b883b4c4ca20ea52b5c37f5c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:44.919552 22427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59571d3897524aa091298aeaa9c13c63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:44.930835 22516 tablet_bootstrap.cc:492] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28: Bootstrap starting.
I20260925 01:16:44.932559 22517 tablet_bootstrap.cc:492] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4: Bootstrap starting.
I20260925 01:16:44.934125 22518 tablet_bootstrap.cc:492] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926: Bootstrap starting.
I20260925 01:16:44.934640 22516 tablet_bootstrap.cc:654] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:44.935953 22516 log.cc:824] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28: Log is configured to *not* fsync() on all Append() calls
I20260925 01:16:44.936101 22517 tablet_bootstrap.cc:654] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:44.937168 22518 tablet_bootstrap.cc:654] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:44.937330 22517 log.cc:824] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4: Log is configured to *not* fsync() on all Append() calls
I20260925 01:16:44.938378 22518 log.cc:824] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926: Log is configured to *not* fsync() on all Append() calls
I20260925 01:16:44.940356 22516 tablet_bootstrap.cc:492] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28: No bootstrap required, opened a new log
I20260925 01:16:44.940725 22516 ts_tablet_manager.cc:1397] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260925 01:16:44.947770 22518 tablet_bootstrap.cc:492] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926: No bootstrap required, opened a new log
I20260925 01:16:44.948079 22518 ts_tablet_manager.cc:1397] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260925 01:16:44.950938 22517 tablet_bootstrap.cc:492] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4: No bootstrap required, opened a new log
I20260925 01:16:44.952834 22517 ts_tablet_manager.cc:1397] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.012s
I20260925 01:16:44.956090 22516 raft_consensus.cc:348] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } }
I20260925 01:16:44.956537 22516 raft_consensus.cc:374] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:44.956657 22516 raft_consensus.cc:729] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0672a4ca904c2782dbe9b1aa061f28, State: Initialized, Role: FOLLOWER
I20260925 01:16:44.957334 22516 consensus_queue.cc:260] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } }
I20260925 01:16:44.957345 22518 raft_consensus.cc:348] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } }
I20260925 01:16:44.957705 22518 raft_consensus.cc:374] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:44.957784 22518 raft_consensus.cc:729] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d0e2fb66b04a509e58367147779926, State: Initialized, Role: FOLLOWER
I20260925 01:16:44.958362 22518 consensus_queue.cc:260] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } }
I20260925 01:16:44.959462 22358 heartbeater.cc:503] Master 127.20.102.190:35757 was elected leader, sending a full tablet report...
I20260925 01:16:44.960111 22518 ts_tablet_manager.cc:1428] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260925 01:16:44.960976 22497 heartbeater.cc:503] Master 127.20.102.190:35757 was elected leader, sending a full tablet report...
I20260925 01:16:44.961596 22516 ts_tablet_manager.cc:1428] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28: Time spent starting tablet: real 0.021s user 0.009s sys 0.009s
I20260925 01:16:44.964161 22517 raft_consensus.cc:348] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } }
I20260925 01:16:44.964516 22517 raft_consensus.cc:374] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:44.964607 22517 raft_consensus.cc:729] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad56dbd458cf4f90a6cc794677996fb4, State: Initialized, Role: FOLLOWER
I20260925 01:16:44.965370 22517 consensus_queue.cc:260] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } }
I20260925 01:16:44.969851 22517 ts_tablet_manager.cc:1428] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4: Time spent starting tablet: real 0.017s user 0.013s sys 0.002s
W20260925 01:16:45.041801 22220 tablet.cc:2369] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:16:45.045565 22359 tablet.cc:2369] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:16:45.061946 22498 tablet.cc:2369] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:16:45.094857 22522 raft_consensus.cc:482] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:16:45.095212 22522 raft_consensus.cc:504] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } }
I20260925 01:16:45.097316 22522 leader_election.cc:288] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0d0e2fb66b04a509e58367147779926 (127.20.102.130:36899), ad56dbd458cf4f90a6cc794677996fb4 (127.20.102.129:40283)
I20260925 01:16:45.103336 22515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:45.098034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:60573 (local address 127.20.102.129:40283)
0925 01:16:45.098242 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:45.098246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:45.098264 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:16:45.098331 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:45.098334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:45.098396 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:45.098542 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:45.098975 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.099730 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.101524 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.102317 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.102529 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:45.102617 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:45.102918 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 01:16:45.103056 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:16:45.103356 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:45.097588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:40283 (local address 127.20.102.131:60573)
0925 01:16:45.097821 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:45.097848 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:45.097979 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:45.098563 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:45.098568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:45.098600 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:45.098818 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:45.098827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.099888 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.099896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:45.101329 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:45.101342 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.102521 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.102528 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:45.102744 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:45.102770 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 01:16:45.102895 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 01:16:45.103052 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 01:16:45.103732 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:45.097839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:45209 (local address 127.20.102.130:36899)
0925 01:16:45.098048 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:45.098055 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:45.098663 (+ 608us) server_negotiation.cc:407] Connection header received
0925 01:16:45.098861 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:45.098866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:45.098933 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:45.099041 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:45.099577 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.100264 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.101968 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.102729 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.102913 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:45.102998 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:45.103310 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 01:16:45.103435 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:16:45.103811 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:45.097746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:36899 (local address 127.20.102.131:45209)
0925 01:16:45.098559 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:45.098573 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:45.098717 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:45.099055 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:45.099059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:45.099080 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:45.099415 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:45.099423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.100409 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.100414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:45.101808 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:45.101820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.102948 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.102954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:45.103106 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:45.103158 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 01:16:45.103274 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 01:16:45.103408 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":1,"thread_start_us":509,"threads_started":1}
I20260925 01:16:45.104568 22169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59571d3897524aa091298aeaa9c13c63" candidate_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad56dbd458cf4f90a6cc794677996fb4" is_pre_election: true
I20260925 01:16:45.104980 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59571d3897524aa091298aeaa9c13c63" candidate_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d0e2fb66b04a509e58367147779926" is_pre_election: true
I20260925 01:16:45.105307 22169 raft_consensus.cc:2454] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0672a4ca904c2782dbe9b1aa061f28 in term 0.
I20260925 01:16:45.105598 22312 raft_consensus.cc:2454] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0672a4ca904c2782dbe9b1aa061f28 in term 0.
I20260925 01:16:45.106169 22379 leader_election.cc:302] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0672a4ca904c2782dbe9b1aa061f28, ad56dbd458cf4f90a6cc794677996fb4; no voters:
I20260925 01:16:45.106622 22522 raft_consensus.cc:2781] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:16:45.106829 22522 raft_consensus.cc:482] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:16:45.106968 22522 raft_consensus.cc:3037] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:45.109417 22522 raft_consensus.cc:504] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } }
I20260925 01:16:45.110154 22522 leader_election.cc:288] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [CANDIDATE]: Term 1 election: Requested vote from peers f0d0e2fb66b04a509e58367147779926 (127.20.102.130:36899), ad56dbd458cf4f90a6cc794677996fb4 (127.20.102.129:40283)
I20260925 01:16:45.110594 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59571d3897524aa091298aeaa9c13c63" candidate_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d0e2fb66b04a509e58367147779926"
I20260925 01:16:45.110715 22169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59571d3897524aa091298aeaa9c13c63" candidate_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad56dbd458cf4f90a6cc794677996fb4"
I20260925 01:16:45.110839 22312 raft_consensus.cc:3037] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:45.110955 22169 raft_consensus.cc:3037] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:45.114302 22312 raft_consensus.cc:2454] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0672a4ca904c2782dbe9b1aa061f28 in term 1.
I20260925 01:16:45.114295 22169 raft_consensus.cc:2454] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0672a4ca904c2782dbe9b1aa061f28 in term 1.
I20260925 01:16:45.114785 22379 leader_election.cc:302] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0672a4ca904c2782dbe9b1aa061f28, ad56dbd458cf4f90a6cc794677996fb4; no voters:
I20260925 01:16:45.115132 22522 raft_consensus.cc:2781] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:45.115741 22522 raft_consensus.cc:686] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 1 LEADER]: Becoming Leader. State: Replica: 1d0672a4ca904c2782dbe9b1aa061f28, State: Running, Role: LEADER
I20260925 01:16:45.116413 22522 consensus_queue.cc:237] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } }
I20260925 01:16:45.121064 22012 catalog_manager.cc:5495] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0672a4ca904c2782dbe9b1aa061f28 (127.20.102.131). New cstate: current_term: 1 leader_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } health_report { overall_health: UNKNOWN } } }
I20260925 01:16:45.155761 22513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:45.150611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55506 (local address 127.20.102.131:33397)
0925 01:16:45.150799 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:45.150803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:45.150957 (+ 154us) server_negotiation.cc:407] Connection header received
0925 01:16:45.151204 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:45.151208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:45.151294 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:45.151412 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:16:45.151758 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.152440 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.153749 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.154158 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.154248 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:45.154707 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:16:45.155275 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:16:45.155378 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:45.155500 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 01:16:45.155564 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:16:45.163784 22169 raft_consensus.cc:1261] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4 [term 1 FOLLOWER]: Refusing update from remote peer 1d0672a4ca904c2782dbe9b1aa061f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:45.163779 22312 raft_consensus.cc:1261] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926 [term 1 FOLLOWER]: Refusing update from remote peer 1d0672a4ca904c2782dbe9b1aa061f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:45.164737 22526 consensus_queue.cc:1035] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:45.165069 22522 consensus_queue.cc:1035] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:45.197707 22011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:16:45.198035 22011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:16:45.201658 22011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e: Sending DeleteTablet for 3 replicas of tablet 59571d3897524aa091298aeaa9c13c63
I20260925 01:16:45.202821 22288 tablet_service.cc:1514] Processing DeleteTablet for tablet 59571d3897524aa091298aeaa9c13c63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:45 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:41262
I20260925 01:16:45.202872 22427 tablet_service.cc:1514] Processing DeleteTablet for tablet 59571d3897524aa091298aeaa9c13c63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:45 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:55504
I20260925 01:16:45.203104 22149 tablet_service.cc:1514] Processing DeleteTablet for tablet 59571d3897524aa091298aeaa9c13c63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:45 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:51406
I20260925 01:16:45.206269 22011 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:16:45.208153 22011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:16:45.215730 22540 tablet_replica.cc:330] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926: stopping tablet replica
I20260925 01:16:45.216212 22540 raft_consensus.cc:2229] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:16:45.218863 22540 raft_consensus.cc:2258] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:45.221060 22426 tablet_service.cc:1467] Processing CreateTablet for tablet 46431a5ca7bd4346986306c5ecf85a27 (DEFAULT_TABLE table=test-table [id=38db78c1b05447f9b17480d1a78bf102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:45.221381 22287 tablet_service.cc:1467] Processing CreateTablet for tablet 46431a5ca7bd4346986306c5ecf85a27 (DEFAULT_TABLE table=test-table [id=38db78c1b05447f9b17480d1a78bf102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:45.221868 22287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46431a5ca7bd4346986306c5ecf85a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:45.222412 22148 tablet_service.cc:1467] Processing CreateTablet for tablet 46431a5ca7bd4346986306c5ecf85a27 (DEFAULT_TABLE table=test-table [id=38db78c1b05447f9b17480d1a78bf102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:45.223032 22148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46431a5ca7bd4346986306c5ecf85a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:45.223110 22540 ts_tablet_manager.cc:1905] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:45.224653 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46431a5ca7bd4346986306c5ecf85a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:45.228662 22518 tablet_bootstrap.cc:492] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926: Bootstrap starting.
I20260925 01:16:45.229702 22517 tablet_bootstrap.cc:492] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4: Bootstrap starting.
I20260925 01:16:45.230367 22518 tablet_bootstrap.cc:654] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:45.231213 22517 tablet_bootstrap.cc:654] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:45.233379 22539 tablet_replica.cc:330] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28: stopping tablet replica
I20260925 01:16:45.233404 22516 tablet_bootstrap.cc:492] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28: Bootstrap starting.
I20260925 01:16:45.233850 22539 raft_consensus.cc:2229] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:16:45.234552 22539 raft_consensus.cc:2258] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:45.235100 22516 tablet_bootstrap.cc:654] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:45.237243 22541 tablet_replica.cc:330] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4: stopping tablet replica
I20260925 01:16:45.237890 22541 raft_consensus.cc:2229] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:16:45.238780 22539 ts_tablet_manager.cc:1905] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:45.240327 22541 raft_consensus.cc:2258] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:45.244973 22517 tablet_bootstrap.cc:492] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4: No bootstrap required, opened a new log
I20260925 01:16:45.245203 22517 ts_tablet_manager.cc:1397] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.005s
I20260925 01:16:45.246122 22518 tablet_bootstrap.cc:492] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926: No bootstrap required, opened a new log
I20260925 01:16:45.246301 22518 ts_tablet_manager.cc:1397] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.008s
I20260925 01:16:45.246083 22517 raft_consensus.cc:348] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.246484 22517 raft_consensus.cc:374] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:45.246903 22517 raft_consensus.cc:729] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad56dbd458cf4f90a6cc794677996fb4, State: Initialized, Role: FOLLOWER
I20260925 01:16:45.247303 22540 ts_tablet_manager.cc:1918] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:16:45.247246 22518 raft_consensus.cc:348] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.247478 22540 log.cc:1196] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/59571d3897524aa091298aeaa9c13c63
I20260925 01:16:45.247510 22518 raft_consensus.cc:374] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:45.247644 22518 raft_consensus.cc:729] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d0e2fb66b04a509e58367147779926, State: Initialized, Role: FOLLOWER
I20260925 01:16:45.247278 22517 consensus_queue.cc:260] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.248085 22540 ts_tablet_manager.cc:1939] T 59571d3897524aa091298aeaa9c13c63 P f0d0e2fb66b04a509e58367147779926: Deleting consensus metadata
I20260925 01:16:45.249094 22518 consensus_queue.cc:260] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.249749 22517 ts_tablet_manager.cc:1428] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260925 01:16:45.250942 22516 tablet_bootstrap.cc:492] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28: No bootstrap required, opened a new log
I20260925 01:16:45.251145 22516 ts_tablet_manager.cc:1397] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.001s
I20260925 01:16:45.251936 22516 raft_consensus.cc:348] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.252202 22516 raft_consensus.cc:374] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:45.252321 22516 raft_consensus.cc:729] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0672a4ca904c2782dbe9b1aa061f28, State: Initialized, Role: FOLLOWER
I20260925 01:16:45.252563 22539 ts_tablet_manager.cc:1918] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:16:45.252612 22516 consensus_queue.cc:260] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.252810 22539 log.cc:1196] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/59571d3897524aa091298aeaa9c13c63
I20260925 01:16:45.253405 22518 ts_tablet_manager.cc:1428] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20260925 01:16:45.253278 22516 ts_tablet_manager.cc:1428] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 01:16:45.256167 21998 catalog_manager.cc:4841] TS f0d0e2fb66b04a509e58367147779926 (127.20.102.130:36899): tablet 59571d3897524aa091298aeaa9c13c63 (table test-table [id=19273c9b883b4c4ca20ea52b5c37f5c5]) successfully deleted
I20260925 01:16:45.256448 22541 ts_tablet_manager.cc:1905] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:45.257745 22539 ts_tablet_manager.cc:1939] T 59571d3897524aa091298aeaa9c13c63 P 1d0672a4ca904c2782dbe9b1aa061f28: Deleting consensus metadata
I20260925 01:16:45.259481 21999 catalog_manager.cc:4841] TS 1d0672a4ca904c2782dbe9b1aa061f28 (127.20.102.131:33397): tablet 59571d3897524aa091298aeaa9c13c63 (table test-table [id=19273c9b883b4c4ca20ea52b5c37f5c5]) successfully deleted
I20260925 01:16:45.262495 22541 ts_tablet_manager.cc:1918] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:16:45.262657 22541 log.cc:1196] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/59571d3897524aa091298aeaa9c13c63
I20260925 01:16:45.263243 22541 ts_tablet_manager.cc:1939] T 59571d3897524aa091298aeaa9c13c63 P ad56dbd458cf4f90a6cc794677996fb4: Deleting consensus metadata
I20260925 01:16:45.264667 21997 catalog_manager.cc:4841] TS ad56dbd458cf4f90a6cc794677996fb4 (127.20.102.129:40283): tablet 59571d3897524aa091298aeaa9c13c63 (table test-table [id=19273c9b883b4c4ca20ea52b5c37f5c5]) successfully deleted
I20260925 01:16:45.649981 22534 raft_consensus.cc:482] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:16:45.650197 22534 raft_consensus.cc:504] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.650915 22534 leader_election.cc:288] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad56dbd458cf4f90a6cc794677996fb4 (127.20.102.129:40283), f0d0e2fb66b04a509e58367147779926 (127.20.102.130:36899)
I20260925 01:16:45.651516 22169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46431a5ca7bd4346986306c5ecf85a27" candidate_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad56dbd458cf4f90a6cc794677996fb4" is_pre_election: true
I20260925 01:16:45.651554 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46431a5ca7bd4346986306c5ecf85a27" candidate_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d0e2fb66b04a509e58367147779926" is_pre_election: true
I20260925 01:16:45.651806 22169 raft_consensus.cc:2454] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0672a4ca904c2782dbe9b1aa061f28 in term 0.
I20260925 01:16:45.651851 22312 raft_consensus.cc:2454] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0672a4ca904c2782dbe9b1aa061f28 in term 0.
I20260925 01:16:45.652299 22379 leader_election.cc:302] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0672a4ca904c2782dbe9b1aa061f28, ad56dbd458cf4f90a6cc794677996fb4; no voters:
I20260925 01:16:45.652626 22534 raft_consensus.cc:2781] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:16:45.652812 22534 raft_consensus.cc:482] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:16:45.652944 22534 raft_consensus.cc:3037] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:45.655277 22534 raft_consensus.cc:504] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.655826 22534 leader_election.cc:288] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [CANDIDATE]: Term 1 election: Requested vote from peers ad56dbd458cf4f90a6cc794677996fb4 (127.20.102.129:40283), f0d0e2fb66b04a509e58367147779926 (127.20.102.130:36899)
I20260925 01:16:45.656498 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46431a5ca7bd4346986306c5ecf85a27" candidate_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d0e2fb66b04a509e58367147779926"
I20260925 01:16:45.656574 22169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46431a5ca7bd4346986306c5ecf85a27" candidate_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad56dbd458cf4f90a6cc794677996fb4"
I20260925 01:16:45.656719 22312 raft_consensus.cc:3037] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:45.656816 22169 raft_consensus.cc:3037] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:45.658746 22312 raft_consensus.cc:2454] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0672a4ca904c2782dbe9b1aa061f28 in term 1.
I20260925 01:16:45.659150 22380 leader_election.cc:302] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0672a4ca904c2782dbe9b1aa061f28, f0d0e2fb66b04a509e58367147779926; no voters:
I20260925 01:16:45.659423 22534 raft_consensus.cc:2781] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:45.659570 22534 raft_consensus.cc:686] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 1 LEADER]: Becoming Leader. State: Replica: 1d0672a4ca904c2782dbe9b1aa061f28, State: Running, Role: LEADER
I20260925 01:16:45.659811 22534 consensus_queue.cc:237] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.660156 22169 raft_consensus.cc:2454] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0672a4ca904c2782dbe9b1aa061f28 in term 1.
I20260925 01:16:45.662459 22011 catalog_manager.cc:5495] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0672a4ca904c2782dbe9b1aa061f28 (127.20.102.131). New cstate: current_term: 1 leader_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } health_report { overall_health: HEALTHY } } }
Sep 25 01:16:45 dist-test-slave-s12s krb5kdc[21967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299001, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:45.724567 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:45.710880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35757 (local address 127.0.0.1:45808)
0925 01:16:45.711649 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:45.711664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:45.711825 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:45.712159 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:45.712164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:45.712874 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:45.713079 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:45.713088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.714157 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.714164 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:45.715087 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:45.715098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.715843 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.715850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:45.716039 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:45.716662 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:45.716830 (+ 168us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:45.719069 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:45.722034 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:45.722047 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:45.722053 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:45.722456 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:45.722814 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:45.722817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:45.722820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:45.722989 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:45.723618 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:45.723624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:45.723771 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 01:16:45.723909 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 01:16:45.724133 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":354,"threads_started":1}
I20260925 01:16:45.724663 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:45.710868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.20.102.190:35757)
0925 01:16:45.711556 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:45.711564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:45.711750 (+ 186us) server_negotiation.cc:407] Connection header received
0925 01:16:45.711942 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:45.711946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:45.712019 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:45.712128 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:45.713251 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.714015 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.715233 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.715684 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.715826 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:45.719245 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:45.719250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:45.719254 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:45.719282 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:45.721903 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:45.722573 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:45.722577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:45.722579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:45.722705 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:45.723128 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:45.723131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:45.723133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:45.723480 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:45.723603 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:45.724010 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 01:16:45.724263 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":499,"thread_start_us":331,"threads_started":1}
I20260925 01:16:45.729794 22011 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:45808:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 01:16:45.730875 22011 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:16:45.737597 22149 tablet_service.cc:1467] Processing CreateTablet for tablet 4344e162626846289d278f41993df572 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72a4dea927044d369a4174a790f89057]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:16:45.737692 22287 tablet_service.cc:1467] Processing CreateTablet for tablet 4344e162626846289d278f41993df572 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72a4dea927044d369a4174a790f89057]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:16:45.738180 22287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4344e162626846289d278f41993df572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:45.738200 22149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4344e162626846289d278f41993df572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:45.738157 22426 tablet_service.cc:1467] Processing CreateTablet for tablet 4344e162626846289d278f41993df572 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72a4dea927044d369a4174a790f89057]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:16:45.738664 22426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4344e162626846289d278f41993df572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:45.744302 22516 tablet_bootstrap.cc:492] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28: Bootstrap starting.
I20260925 01:16:45.745394 22517 tablet_bootstrap.cc:492] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4: Bootstrap starting.
I20260925 01:16:45.745455 22518 tablet_bootstrap.cc:492] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926: Bootstrap starting.
I20260925 01:16:45.745949 22516 tablet_bootstrap.cc:654] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:45.747174 22518 tablet_bootstrap.cc:654] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:45.747253 22517 tablet_bootstrap.cc:654] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:45.748785 22516 tablet_bootstrap.cc:492] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28: No bootstrap required, opened a new log
I20260925 01:16:45.748962 22516 ts_tablet_manager.cc:1397] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 01:16:45.749378 22518 tablet_bootstrap.cc:492] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926: No bootstrap required, opened a new log
I20260925 01:16:45.749553 22518 ts_tablet_manager.cc:1397] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 01:16:45.749784 22516 raft_consensus.cc:348] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.750041 22516 raft_consensus.cc:374] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:45.750128 22516 raft_consensus.cc:729] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0672a4ca904c2782dbe9b1aa061f28, State: Initialized, Role: FOLLOWER
I20260925 01:16:45.750319 22518 raft_consensus.cc:348] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.750588 22518 raft_consensus.cc:374] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:45.750667 22518 raft_consensus.cc:729] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d0e2fb66b04a509e58367147779926, State: Initialized, Role: FOLLOWER
I20260925 01:16:45.750569 22516 consensus_queue.cc:260] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.750896 22518 consensus_queue.cc:260] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.750993 22534 tablet_replica.cc:439] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } }
I20260925 01:16:45.751293 22534 tablet_replica.cc:442] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:16:45.751444 22518 ts_tablet_manager.cc:1428] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 01:16:45.751962 22523 tablet_replica.cc:439] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } }
I20260925 01:16:45.752229 22523 tablet_replica.cc:442] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:16:45.752297 22516 ts_tablet_manager.cc:1428] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 01:16:45.753123 22517 tablet_bootstrap.cc:492] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4: No bootstrap required, opened a new log
I20260925 01:16:45.753322 22517 ts_tablet_manager.cc:1397] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260925 01:16:45.753979 22517 raft_consensus.cc:348] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.754230 22517 raft_consensus.cc:374] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:45.754779 22517 raft_consensus.cc:729] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad56dbd458cf4f90a6cc794677996fb4, State: Initialized, Role: FOLLOWER
I20260925 01:16:45.755115 22517 consensus_queue.cc:260] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.756248 22517 ts_tablet_manager.cc:1428] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 01:16:45.756191 22556 tablet_replica.cc:439] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } }
I20260925 01:16:45.756475 22556 tablet_replica.cc:442] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:16:45.897917 22534 raft_consensus.cc:482] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:16:45.898108 22534 raft_consensus.cc:504] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.898710 22534 leader_election.cc:288] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad56dbd458cf4f90a6cc794677996fb4 (127.20.102.129:40283), f0d0e2fb66b04a509e58367147779926 (127.20.102.130:36899)
I20260925 01:16:45.899240 22169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4344e162626846289d278f41993df572" candidate_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad56dbd458cf4f90a6cc794677996fb4" is_pre_election: true
I20260925 01:16:45.899282 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4344e162626846289d278f41993df572" candidate_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d0e2fb66b04a509e58367147779926" is_pre_election: true
I20260925 01:16:45.899510 22169 raft_consensus.cc:2454] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0672a4ca904c2782dbe9b1aa061f28 in term 0.
I20260925 01:16:45.899555 22312 raft_consensus.cc:2454] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d0672a4ca904c2782dbe9b1aa061f28 in term 0.
I20260925 01:16:45.899945 22379 leader_election.cc:302] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0672a4ca904c2782dbe9b1aa061f28, ad56dbd458cf4f90a6cc794677996fb4; no voters:
I20260925 01:16:45.900211 22534 raft_consensus.cc:2781] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:16:45.900318 22534 raft_consensus.cc:482] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:16:45.900424 22534 raft_consensus.cc:3037] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:45.902490 22534 raft_consensus.cc:504] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.903041 22534 leader_election.cc:288] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [CANDIDATE]: Term 1 election: Requested vote from peers ad56dbd458cf4f90a6cc794677996fb4 (127.20.102.129:40283), f0d0e2fb66b04a509e58367147779926 (127.20.102.130:36899)
I20260925 01:16:45.903501 22169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4344e162626846289d278f41993df572" candidate_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad56dbd458cf4f90a6cc794677996fb4"
I20260925 01:16:45.903502 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4344e162626846289d278f41993df572" candidate_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0d0e2fb66b04a509e58367147779926"
I20260925 01:16:45.903746 22169 raft_consensus.cc:3037] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:45.903761 22312 raft_consensus.cc:3037] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:45.906618 22312 raft_consensus.cc:2454] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0672a4ca904c2782dbe9b1aa061f28 in term 1.
I20260925 01:16:45.906617 22169 raft_consensus.cc:2454] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d0672a4ca904c2782dbe9b1aa061f28 in term 1.
I20260925 01:16:45.907030 22380 leader_election.cc:302] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0672a4ca904c2782dbe9b1aa061f28, f0d0e2fb66b04a509e58367147779926; no voters:
I20260925 01:16:45.907341 22534 raft_consensus.cc:2781] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:45.907526 22534 raft_consensus.cc:686] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 1 LEADER]: Becoming Leader. State: Replica: 1d0672a4ca904c2782dbe9b1aa061f28, State: Running, Role: LEADER
I20260925 01:16:45.907786 22534 consensus_queue.cc:237] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } }
I20260925 01:16:45.909471 22522 tablet_replica.cc:439] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28: TxnStatusTablet state changed. Reason: New leader 1d0672a4ca904c2782dbe9b1aa061f28. Latest consensus state: current_term: 1 leader_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } }
I20260925 01:16:45.909711 22522 tablet_replica.cc:442] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:16:45.910251 22558 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 01:16:45.910501 22558 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 01:16:45.911064 22012 catalog_manager.cc:5495] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d0672a4ca904c2782dbe9b1aa061f28 (127.20.102.131). New cstate: current_term: 1 leader_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } health_report { overall_health: HEALTHY } } }
I20260925 01:16:45.945204 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:45.939105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:33397 (local address 127.0.0.1:55518)
0925 01:16:45.939427 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:45.939455 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:45.939599 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:45.940101 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:45.940104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:45.940115 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:16:45.940278 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:45.940285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.941932 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.941938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:45.943079 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:45.943091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.944070 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.944074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:45.944243 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:45.944304 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:16:45.944763 (+ 459us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:16:45.944781 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 01:16:45.944847 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 01:16:45.944958 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 01:16:45.945382 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:45.939068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55518 (local address 127.20.102.131:33397)
0925 01:16:45.939779 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:45.939786 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:45.939807 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:16:45.939873 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:45.939879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:45.939961 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:45.940080 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:16:45.940459 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.941746 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.943238 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:45.943910 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:45.944059 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:45.944491 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:16:45.944633 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:16:45.944766 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:45.944918 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 01:16:45.945000 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":439,"threads_started":1}
I20260925 01:16:45.949471 22169 raft_consensus.cc:1261] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4 [term 1 FOLLOWER]: Refusing update from remote peer 1d0672a4ca904c2782dbe9b1aa061f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:45.949622 22312 raft_consensus.cc:1261] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926 [term 1 FOLLOWER]: Refusing update from remote peer 1d0672a4ca904c2782dbe9b1aa061f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:45.950263 22522 consensus_queue.cc:1035] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:45.950753 22534 consensus_queue.cc:1035] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:45.953866 22556 tablet_replica.cc:439] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4: TxnStatusTablet state changed. Reason: New leader 1d0672a4ca904c2782dbe9b1aa061f28. Latest consensus state: current_term: 1 leader_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } }
I20260925 01:16:45.954111 22556 tablet_replica.cc:442] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:16:45.956347 22562 mvcc.cc:205] Tried to move back new op lower bound from 7333064728362233856 to 7333064728200540160. Current Snapshot: MvccSnapshot[applied={T|T < 7333064728362233856}]
I20260925 01:16:45.966251 22556 tablet_replica.cc:439] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } }
I20260925 01:16:45.966540 22556 tablet_replica.cc:442] T 4344e162626846289d278f41993df572 P ad56dbd458cf4f90a6cc794677996fb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:16:45.969568 22534 tablet_replica.cc:439] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } }
I20260925 01:16:45.969884 22534 tablet_replica.cc:442] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:16:45.974351 22523 tablet_replica.cc:439] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926: TxnStatusTablet state changed. Reason: New leader 1d0672a4ca904c2782dbe9b1aa061f28. Latest consensus state: current_term: 1 leader_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } }
I20260925 01:16:45.974617 22523 tablet_replica.cc:442] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:16:45.975281 22522 tablet_replica.cc:439] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } }
I20260925 01:16:45.975529 22522 tablet_replica.cc:442] T 4344e162626846289d278f41993df572 P 1d0672a4ca904c2782dbe9b1aa061f28: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:16:45.991029 22523 tablet_replica.cc:439] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 } } peers { permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 } } peers { permanent_uuid: "1d0672a4ca904c2782dbe9b1aa061f28" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33397 } } }
I20260925 01:16:45.991343 22523 tablet_replica.cc:442] T 4344e162626846289d278f41993df572 P f0d0e2fb66b04a509e58367147779926: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:16:46.006172 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:45.999076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:40789 (local address 127.20.102.131:33397)
0925 01:16:45.999277 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:45.999288 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:45.999950 (+ 662us) server_negotiation.cc:407] Connection header received
0925 01:16:46.000212 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:46.000225 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:46.000298 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:46.000408 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:16:46.001169 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:46.002079 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:46.003953 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:46.004533 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:46.004744 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:46.005235 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:16:46.005406 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:16:46.005526 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:46.005685 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 01:16:46.005768 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:16:46.006448 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:45.999139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:33397 (local address 127.20.102.131:40789)
0925 01:16:45.999818 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:45.999838 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:46.000042 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:46.000419 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:46.000425 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:46.000439 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:16:46.000982 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:46.000995 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:46.002234 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:46.002243 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:46.003758 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:46.003810 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:46.004773 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:46.004779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:46.004954 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:46.005010 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:16:46.005526 (+ 516us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:16:46.005539 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 01:16:46.005653 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 01:16:46.005808 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":27,"thread_start_us":366,"threads_started":1}
I20260925 01:16:46.020153 22169 raft_consensus.cc:1261] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4 [term 1 FOLLOWER]: Refusing update from remote peer 1d0672a4ca904c2782dbe9b1aa061f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:46.020161 22312 raft_consensus.cc:1261] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926 [term 1 FOLLOWER]: Refusing update from remote peer 1d0672a4ca904c2782dbe9b1aa061f28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:46.020805 22534 consensus_queue.cc:1035] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0d0e2fb66b04a509e58367147779926" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:46.021311 22522 consensus_queue.cc:1035] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad56dbd458cf4f90a6cc794677996fb4" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:46.037498 22563 mvcc.cc:205] Tried to move back new op lower bound from 7333064728652152832 to 7333064727184166912. Current Snapshot: MvccSnapshot[applied={T|T < 7333064728652152832}]
I20260925 01:16:46.042210 22564 mvcc.cc:205] Tried to move back new op lower bound from 7333064728652152832 to 7333064727184166912. Current Snapshot: MvccSnapshot[applied={T|T < 7333064728652152832}]
I20260925 01:16:46.093190 22011 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:16:46.093390 22011 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:16:46.096302 22011 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48dcaa2b79914f8fb5548bc085de626e: Sending DeleteTablet for 3 replicas of tablet 46431a5ca7bd4346986306c5ecf85a27
I20260925 01:16:46.097026 22149 tablet_service.cc:1514] Processing DeleteTablet for tablet 46431a5ca7bd4346986306c5ecf85a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:46 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:51406
I20260925 01:16:46.097153 22287 tablet_service.cc:1514] Processing DeleteTablet for tablet 46431a5ca7bd4346986306c5ecf85a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:46 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:41262
I20260925 01:16:46.097355 22427 tablet_service.cc:1514] Processing DeleteTablet for tablet 46431a5ca7bd4346986306c5ecf85a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:46 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:55504
I20260925 01:16:46.108134 22578 tablet_replica.cc:330] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926: stopping tablet replica
I20260925 01:16:46.108394 22578 raft_consensus.cc:2229] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:16:46.108898 22578 raft_consensus.cc:2258] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:46.109615 22577 tablet_replica.cc:330] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4: stopping tablet replica
I20260925 01:16:46.110105 22578 ts_tablet_manager.cc:1905] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:46.110049 22577 raft_consensus.cc:2229] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:16:46.110632 22577 raft_consensus.cc:2258] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:46.111940 22577 ts_tablet_manager.cc:1905] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:46.117475 22578 ts_tablet_manager.cc:1918] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:16:46.117610 22578 log.cc:1196] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/46431a5ca7bd4346986306c5ecf85a27
I20260925 01:16:46.118135 22578 ts_tablet_manager.cc:1939] T 46431a5ca7bd4346986306c5ecf85a27 P f0d0e2fb66b04a509e58367147779926: Deleting consensus metadata
I20260925 01:16:46.119203 21998 catalog_manager.cc:4841] TS f0d0e2fb66b04a509e58367147779926 (127.20.102.130:36899): tablet 46431a5ca7bd4346986306c5ecf85a27 (table test-table [id=38db78c1b05447f9b17480d1a78bf102]) successfully deleted
I20260925 01:16:46.120923 22577 ts_tablet_manager.cc:1918] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:16:46.121165 22577 log.cc:1196] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/46431a5ca7bd4346986306c5ecf85a27
I20260925 01:16:46.121716 22577 ts_tablet_manager.cc:1939] T 46431a5ca7bd4346986306c5ecf85a27 P ad56dbd458cf4f90a6cc794677996fb4: Deleting consensus metadata
I20260925 01:16:46.122148 22579 tablet_replica.cc:330] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28: stopping tablet replica
I20260925 01:16:46.122361 22579 raft_consensus.cc:2229] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:16:46.122823 22579 raft_consensus.cc:2258] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:46.122996 21997 catalog_manager.cc:4841] TS ad56dbd458cf4f90a6cc794677996fb4 (127.20.102.129:40283): tablet 46431a5ca7bd4346986306c5ecf85a27 (table test-table [id=38db78c1b05447f9b17480d1a78bf102]) successfully deleted
I20260925 01:16:46.123912 22579 ts_tablet_manager.cc:1905] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:46.129686 22579 ts_tablet_manager.cc:1918] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:16:46.129822 22579 log.cc:1196] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/46431a5ca7bd4346986306c5ecf85a27
I20260925 01:16:46.130388 22579 ts_tablet_manager.cc:1939] T 46431a5ca7bd4346986306c5ecf85a27 P 1d0672a4ca904c2782dbe9b1aa061f28: Deleting consensus metadata
I20260925 01:16:46.131552 21999 catalog_manager.cc:4841] TS 1d0672a4ca904c2782dbe9b1aa061f28 (127.20.102.131:33397): tablet 46431a5ca7bd4346986306c5ecf85a27 (table test-table [id=38db78c1b05447f9b17480d1a78bf102]) successfully deleted
Sep 25 01:16:46 dist-test-slave-s12s krb5kdc[21967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:16:46.140348 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:46.118205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51408 (local address 127.20.102.129:40283)
0925 01:16:46.121191 (+ 2986us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:46.121198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:46.121907 (+ 709us) server_negotiation.cc:407] Connection header received
0925 01:16:46.122188 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:46.122195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:46.122277 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:46.122357 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:46.124367 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:46.125646 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:46.127541 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:46.128214 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:46.128338 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:46.133556 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:46.133575 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:46.133599 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:46.133644 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:46.137455 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:46.138169 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:46.138178 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:46.138182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:46.138347 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:46.138709 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:46.138714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:46.138717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:46.139139 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:46.139445 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:46.139691 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 01:16:46.139933 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2843,"thread_start_us":2691,"threads_started":1}
W20260925 01:16:46.141167 22214 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51408
I20260925 01:16:46.156327 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:46.144159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45820 (local address 127.20.102.190:35757)
0925 01:16:46.144357 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:46.144366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:46.144621 (+ 255us) server_negotiation.cc:407] Connection header received
0925 01:16:46.144836 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:46.144840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:46.144903 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:46.145017 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:46.146214 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:46.146943 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:46.148220 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:46.148812 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:46.148957 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:46.151172 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:46.151178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:46.151183 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:46.151223 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:46.153810 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:46.154415 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:46.154422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:46.154425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:46.154555 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:46.154912 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:46.154918 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:46.154921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:46.155206 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:46.155468 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:46.155794 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 01:16:46.156030 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 01:16:46.156988 22011 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45820
I20260925 01:16:46.160559 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 22083
I20260925 01:16:46.179446 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:46.161350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51422 (local address 127.20.102.129:40283)
0925 01:16:46.161608 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:46.161618 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:46.166569 (+ 4951us) server_negotiation.cc:407] Connection header received
0925 01:16:46.166750 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:46.166756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:46.166859 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:46.166959 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:46.168204 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:46.169266 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:46.170438 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:46.171156 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:46.171281 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:46.173561 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:46.173568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:46.173594 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:46.173634 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:46.176888 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:46.177465 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:46.177470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:46.177473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:46.177631 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:46.178009 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:46.178014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:46.178017 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:46.178330 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:46.178574 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:46.178803 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 01:16:46.179086 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 01:16:46.180028 22214 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51422
W20260925 01:16:46.180600 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 22083: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:16:46.180830 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 22083
I20260925 01:16:46.204854 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 22223
Sep 25 01:16:46 dist-test-slave-s12s krb5kdc[21967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:16:46.220525 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:46.205676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41264 (local address 127.20.102.130:36899)
0925 01:16:46.206232 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:46.206237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:46.206251 (+ 14us) server_negotiation.cc:407] Connection header received
0925 01:16:46.206313 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:46.206317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:46.206375 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:46.206519 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:46.207729 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:46.208509 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:46.209808 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:46.210349 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:46.210517 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:46.214676 (+ 4159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:46.214691 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:46.214708 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:46.214739 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:46.217763 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:46.218423 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:46.218431 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:46.218433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:46.218590 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:46.218996 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:46.219003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:46.219008 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:46.219337 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:46.219671 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:46.219941 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 01:16:46.220175 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":278,"threads_started":1}
W20260925 01:16:46.221377 22353 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41264
W20260925 01:16:46.222047 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 22223: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:16:46.222187 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 22223
I20260925 01:16:46.246066 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 22362
Sep 25 01:16:46 dist-test-slave-s12s krb5kdc[21967](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:16:46.263068 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:46.246920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.20.102.131:33397)
0925 01:16:46.247126 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:46.247132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:46.247155 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:16:46.247305 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:46.247310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:46.247416 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:46.247527 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:46.248745 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:46.249919 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:46.251182 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:46.251888 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:46.252025 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:46.256899 (+ 4874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:46.256914 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:46.256956 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:46.256999 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:46.260046 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:46.261054 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:46.261061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:46.261063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:46.261210 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:46.261597 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:46.261600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:46.261602 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:46.261968 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:46.262228 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:46.262598 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 01:16:46.262750 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 01:16:46.263857 22492 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55522
W20260925 01:16:46.264564 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 22362: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:16:46.264768 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 22362
I20260925 01:16:46.288830 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 21980
I20260925 01:16:46.290020 22052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:16:46.404098 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 21980
2026-09-25T01:16:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (8223 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:16:46 dist-test-slave-s12s krb5kdc[22590](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:16:46 dist-test-slave-s12s krb5kdc[22590](info): set up 2 sockets
Sep 25 01:16:46 dist-test-slave-s12s krb5kdc[22590](info): commencing operation
krb5kdc: starting...
W20260925 01:16:47.504904 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:16:47 dist-test-slave-s12s krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:16:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:16:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:16:47.688560 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:33339
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:33339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:48.001003 22603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:48.001327 22603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:48.001495 22603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:48.008561 22603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:16:48.008723 22603 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 01:16:48.008824 22603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:48.008940 22603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:48.009037 22603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:16:48.009125 22603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:16:48.018658 22603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:33339
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:33339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:48.019893 22603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:48.021224 22603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:48.033135 22608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:48.033185 22609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:48.033329 22611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:48.034026 22603 server_base.cc:1034] running on GCE node
I20260925 01:16:48.037451 22603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:48.038868 22603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:48.040170 22603 hybrid_clock.cc:648] HybridClock initialized: now 1790299008040087 us; error 44 us; skew 500 ppm
I20260925 01:16:48.040531 22603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:48 dist-test-slave-s12s krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299008, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:48.048339 22603 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:16:48.049875 22603 webserver.cc:460] Webserver started at http://127.20.102.190:38025/ using document root <none> and password file <none>
I20260925 01:16:48.050596 22603 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:48.050735 22603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:48.051069 22603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:48.054209 22603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "a5aad964d4d540818b37d547fa641645"
format_stamp: "Formatted at 2026-09-25 01:16:48 on dist-test-slave-s12s"
server_key: "e711ed3df32c7466e1eeb213d9a244f1"
server_key_iv: "91e724ed0ffbf93411512180e7233600"
server_key_version: "encryptionkey@0"
I20260925 01:16:48.055032 22603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "a5aad964d4d540818b37d547fa641645"
format_stamp: "Formatted at 2026-09-25 01:16:48 on dist-test-slave-s12s"
server_key: "e711ed3df32c7466e1eeb213d9a244f1"
server_key_iv: "91e724ed0ffbf93411512180e7233600"
server_key_version: "encryptionkey@0"
I20260925 01:16:48.060295 22603 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 01:16:48.063849 22618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:48.065156 22603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 01:16:48.065325 22603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "a5aad964d4d540818b37d547fa641645"
format_stamp: "Formatted at 2026-09-25 01:16:48 on dist-test-slave-s12s"
server_key: "e711ed3df32c7466e1eeb213d9a244f1"
server_key_iv: "91e724ed0ffbf93411512180e7233600"
server_key_version: "encryptionkey@0"
I20260925 01:16:48.065546 22603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:48.084453 22603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:48.088637 22603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:48.089000 22603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:48.110098 22603 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:33339
I20260925 01:16:48.110131 22679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:33339 every 8 connection(s)
I20260925 01:16:48.111347 22603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:16:48.113335 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 22603
I20260925 01:16:48.113561 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:16:48.114387 20890 external_mini_cluster.cc:1428] Setting key cd3bc717d9065e4ccbc49839f3886edb
I20260925 01:16:48.115713 22680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:48.126816 22680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645: Bootstrap starting.
Sep 25 01:16:48 dist-test-slave-s12s krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:48.130410 22680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:48.131794 22680 log.cc:824] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645: Log is configured to *not* fsync() on all Append() calls
I20260925 01:16:48.134770 22680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645: No bootstrap required, opened a new log
I20260925 01:16:48.136554 22683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:48.116461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.20.102.190:33339)
0925 01:16:48.117228 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:48.117238 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:48.117277 (+ 39us) server_negotiation.cc:407] Connection header received
0925 01:16:48.118778 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:48.118799 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:48.119174 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:48.119557 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:48.121057 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:48.122339 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:48.123560 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:48.124254 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:48.124456 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:48.128843 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:48.128851 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:48.128867 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:48.128900 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:48.132590 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:48.133307 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:48.133317 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:48.133330 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:48.133515 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:48.133903 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:48.133908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:48.133913 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:48.134408 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:48.134796 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:48.135147 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 01:16:48.135590 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":367,"threads_started":1}
I20260925 01:16:48.142606 22680 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5aad964d4d540818b37d547fa641645" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33339 } }
I20260925 01:16:48.143121 22680 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:48.143232 22680 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5aad964d4d540818b37d547fa641645, State: Initialized, Role: FOLLOWER
I20260925 01:16:48.143889 22680 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5aad964d4d540818b37d547fa641645" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33339 } }
I20260925 01:16:48.144119 22680 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:16:48.144275 22680 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:16:48.144446 22680 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:48.146615 22680 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5aad964d4d540818b37d547fa641645" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33339 } }
I20260925 01:16:48.147128 22680 leader_election.cc:302] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5aad964d4d540818b37d547fa641645; no voters:
I20260925 01:16:48.147720 22680 leader_election.cc:288] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:16:48.147915 22685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:48.148814 22685 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [term 1 LEADER]: Becoming Leader. State: Replica: a5aad964d4d540818b37d547fa641645, State: Running, Role: LEADER
I20260925 01:16:48.149348 22685 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5aad964d4d540818b37d547fa641645" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33339 } }
I20260925 01:16:48.149830 22680 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:16:48.152253 22687 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5aad964d4d540818b37d547fa641645. Latest consensus state: current_term: 1 leader_uuid: "a5aad964d4d540818b37d547fa641645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5aad964d4d540818b37d547fa641645" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33339 } } }
I20260925 01:16:48.152654 22686 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5aad964d4d540818b37d547fa641645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5aad964d4d540818b37d547fa641645" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33339 } } }
I20260925 01:16:48.152999 22687 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [sys.catalog]: This master's current role is: LEADER
I20260925 01:16:48.153012 22686 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645 [sys.catalog]: This master's current role is: LEADER
I20260925 01:16:48.157716 22694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:16:48.162431 22694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:16:48.170586 22694 catalog_manager.cc:1347] Generated new cluster ID: 2acd14b9132b421bb51df91c3bce56ba
I20260925 01:16:48.170693 22694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:16:48.184069 22694 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:16:48.185886 22694 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:16:48.195564 22694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645: Generated new TSK 0
I20260925 01:16:48.196607 22694 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:16:48.248131 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:33339
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:48.581936 22706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:48.582219 22706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:48.582428 22706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:48.589574 22706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:48.589713 22706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:48.589884 22706 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 01:16:48.589977 22706 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:16:48.590123 22706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:16:48.600005 22706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.102.190:33339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:48.601429 22706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:48.602806 22706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:48.616575 22712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:48.616637 22711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:48.616451 22714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:48.617473 22706 server_base.cc:1034] running on GCE node
I20260925 01:16:48.618008 22706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:48.618862 22706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:48.620077 22706 hybrid_clock.cc:648] HybridClock initialized: now 1790299008620041 us; error 29 us; skew 500 ppm
I20260925 01:16:48.620409 22706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:48 dist-test-slave-s12s krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299008, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:48.623512 22706 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:16:48.624979 22706 webserver.cc:460] Webserver started at http://127.20.102.129:39331/ using document root <none> and password file <none>
I20260925 01:16:48.625576 22706 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:48.625669 22706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:48.625962 22706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:48.629841 22706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "23074871eef248bab79f492dbc710aa5"
format_stamp: "Formatted at 2026-09-25 01:16:48 on dist-test-slave-s12s"
server_key: "89bd14f52548993b8eefca02ede72f05"
server_key_iv: "ca8aafba741f647e065c1d74311be3bd"
server_key_version: "encryptionkey@0"
I20260925 01:16:48.630522 22706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "23074871eef248bab79f492dbc710aa5"
format_stamp: "Formatted at 2026-09-25 01:16:48 on dist-test-slave-s12s"
server_key: "89bd14f52548993b8eefca02ede72f05"
server_key_iv: "ca8aafba741f647e065c1d74311be3bd"
server_key_version: "encryptionkey@0"
I20260925 01:16:48.635422 22706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:16:48.638944 22721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:48.640235 22706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:16:48.640417 22706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "23074871eef248bab79f492dbc710aa5"
format_stamp: "Formatted at 2026-09-25 01:16:48 on dist-test-slave-s12s"
server_key: "89bd14f52548993b8eefca02ede72f05"
server_key_iv: "ca8aafba741f647e065c1d74311be3bd"
server_key_version: "encryptionkey@0"
I20260925 01:16:48.640630 22706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:48.667840 22706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:48.671868 22706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:48.672166 22706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:48.674780 22706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:48.674912 22706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:48.675060 22706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:48.675155 22706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:16:48 dist-test-slave-s12s krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299008, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:48.696476 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:48.675197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:48217 (local address 127.20.102.190:33339)
0925 01:16:48.675680 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:48.675686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:48.677034 (+ 1348us) server_negotiation.cc:407] Connection header received
0925 01:16:48.679085 (+ 2051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:48.679091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:48.679179 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:48.679307 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:48.681519 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:48.682575 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:48.685112 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:48.685561 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:48.685699 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:48.690636 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:48.690642 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:48.690645 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:48.690678 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:48.693475 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:48.694172 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:48.694178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:48.694183 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:48.694301 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:48.694733 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:48.694738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:48.694742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:48.695017 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:48.695183 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:48.695863 (+ 680us) server_negotiation.cc:299] Negotiation successful
0925 01:16:48.696077 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":228,"threads_started":1}
I20260925 01:16:48.697412 22731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:48.675935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33339 (local address 127.20.102.129:48217)
0925 01:16:48.676852 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:48.676896 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:48.678788 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:48.679540 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:48.679551 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:48.680288 (+ 737us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:48.681253 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:48.681286 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:48.682733 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:48.682744 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:48.684957 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:48.684969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:48.685801 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:48.685808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:48.686156 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:48.686969 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:48.687010 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:48.690387 (+ 3377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:48.693621 (+ 3234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:48.693629 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:48.693644 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:48.694046 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:48.694418 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:48.694423 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:48.694427 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:48.694613 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:48.695165 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:48.695184 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:48.695497 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 01:16:48.695748 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 01:16:48.696232 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":399,"threads_started":1}
I20260925 01:16:48.717296 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:48.703318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:41859 (local address 127.20.102.190:33339)
0925 01:16:48.703469 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:48.703473 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:48.704115 (+ 642us) server_negotiation.cc:407] Connection header received
0925 01:16:48.704313 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:48.704317 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:48.704385 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:48.704482 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:48.705696 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:48.706609 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:48.708539 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:48.709185 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:48.709320 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:48.710997 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:48.711004 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:48.711009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:48.711054 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:48.713754 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:48.714685 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:48.714688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:48.714691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:48.714832 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:48.715443 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:48.715448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:48.715451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:48.715883 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:48.716049 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:48.716723 (+ 674us) server_negotiation.cc:299] Negotiation successful
0925 01:16:48.716987 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:16:48.717706 22775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:48.703376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33339 (local address 127.20.102.129:41859)
0925 01:16:48.704009 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:48.704024 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:48.704183 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:48.704506 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:48.704510 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:48.705055 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:48.705558 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:48.705570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:48.706749 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:48.706754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:48.708362 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:48.708377 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:48.709394 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:48.709400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:48.709568 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:48.710156 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:48.710172 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:48.710823 (+ 651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:48.713934 (+ 3111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:48.713943 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:48.713949 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:48.714525 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:48.714987 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:48.714993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:48.714999 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:48.715294 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:48.716090 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:48.716099 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:48.716350 (+ 251us) client_negotiation.cc:769] Sending connection context
0925 01:16:48.716589 (+ 239us) client_negotiation.cc:240] Negotiation successful
0925 01:16:48.717059 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":367,"threads_started":1}
I20260925 01:16:48.731077 22706 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:34703
I20260925 01:16:48.731158 22841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:34703 every 8 connection(s)
I20260925 01:16:48.732313 22706 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:16:48.734368 22842 heartbeater.cc:347] Connected to a master server at 127.20.102.190:33339
I20260925 01:16:48.734589 22842 heartbeater.cc:464] Registering TS with master...
I20260925 01:16:48.735260 22842 heartbeater.cc:511] Master 127.20.102.190:33339 requested a full tablet report, sending...
I20260925 01:16:48.736891 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 22706
I20260925 01:16:48.737123 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:16:48.737468 22635 ts_manager.cc:194] Registered new tserver with Master: 23074871eef248bab79f492dbc710aa5 (127.20.102.129:34703)
I20260925 01:16:48.737936 20890 external_mini_cluster.cc:1428] Setting key a3973edf0f62b311a4c5e028c7cd052f
I20260925 01:16:48.739521 22635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:48217
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:16:48.780133 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:33339
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:49.096073 22846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:49.096313 22846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:49.096490 22846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:49.103724 22846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:49.103816 22846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:49.104001 22846 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 01:16:49.104077 22846 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:16:49.104198 22846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:16:49.113636 22846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.102.190:33339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:49.114835 22846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:49.116153 22846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:49.129045 22851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:49.129129 22852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:49.129158 22854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:49.129889 22846 server_base.cc:1034] running on GCE node
I20260925 01:16:49.130402 22846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:49.131172 22846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:49.132371 22846 hybrid_clock.cc:648] HybridClock initialized: now 1790299009132331 us; error 24 us; skew 500 ppm
I20260925 01:16:49.132638 22846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:49 dist-test-slave-s12s krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299009, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:49.135982 22846 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:16:49.137568 22846 webserver.cc:460] Webserver started at http://127.20.102.130:44967/ using document root <none> and password file <none>
I20260925 01:16:49.138281 22846 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:49.138412 22846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:49.138684 22846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:49.141500 22846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "ce342ad910b04a07b732680ce4ab1e08"
format_stamp: "Formatted at 2026-09-25 01:16:49 on dist-test-slave-s12s"
server_key: "8f3e5fa7ccf8d45e45f07869d80335c0"
server_key_iv: "4c9829c8b498f046e9bf1273f2dbd4c9"
server_key_version: "encryptionkey@0"
I20260925 01:16:49.142258 22846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "ce342ad910b04a07b732680ce4ab1e08"
format_stamp: "Formatted at 2026-09-25 01:16:49 on dist-test-slave-s12s"
server_key: "8f3e5fa7ccf8d45e45f07869d80335c0"
server_key_iv: "4c9829c8b498f046e9bf1273f2dbd4c9"
server_key_version: "encryptionkey@0"
I20260925 01:16:49.147382 22846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:16:49.150918 22861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:49.152231 22846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:16:49.152384 22846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "ce342ad910b04a07b732680ce4ab1e08"
format_stamp: "Formatted at 2026-09-25 01:16:49 on dist-test-slave-s12s"
server_key: "8f3e5fa7ccf8d45e45f07869d80335c0"
server_key_iv: "4c9829c8b498f046e9bf1273f2dbd4c9"
server_key_version: "encryptionkey@0"
I20260925 01:16:49.152626 22846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:49.171331 22846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:49.175644 22846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:49.175946 22846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:49.178560 22846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:49.178699 22846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:49.178826 22846 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:49.178917 22846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:16:49 dist-test-slave-s12s krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299009, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:49.204129 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.179310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:34997 (local address 127.20.102.190:33339)
0925 01:16:49.179498 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:49.179505 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:49.180915 (+ 1410us) server_negotiation.cc:407] Connection header received
0925 01:16:49.182953 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:49.182960 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:49.183092 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:49.183202 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:49.185482 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.186840 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.189383 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.190058 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.190214 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.196135 (+ 5921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:49.196144 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:49.196149 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:49.196193 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:49.200193 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:49.201329 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:49.201336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:49.201339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:49.201520 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:49.202220 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:49.202228 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:49.202233 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:49.202589 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:49.202753 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:49.203522 (+ 769us) server_negotiation.cc:299] Negotiation successful
0925 01:16:49.203780 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 01:16:49.205555 22870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.179828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33339 (local address 127.20.102.130:34997)
0925 01:16:49.180720 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:49.180774 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:49.182672 (+ 1898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:49.183458 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:49.183472 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:49.184324 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:49.185250 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:49.185290 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.187057 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.187064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:49.189234 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:49.189246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.190408 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.190423 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:49.190810 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.191822 (+ 1012us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:49.191867 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:49.195779 (+ 3912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:49.200377 (+ 4598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:49.200389 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:49.200414 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:49.201201 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:49.201666 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:49.201677 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:49.201683 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:49.202053 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:49.202728 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:49.202736 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:49.203135 (+ 399us) client_negotiation.cc:769] Sending connection context
0925 01:16:49.203392 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 01:16:49.203971 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":382,"threads_started":1}
I20260925 01:16:49.227128 22846 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:38731
I20260925 01:16:49.227281 22980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:38731 every 8 connection(s)
I20260925 01:16:49.228526 22846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:16:49.231204 22981 heartbeater.cc:347] Connected to a master server at 127.20.102.190:33339
I20260925 01:16:49.231443 22981 heartbeater.cc:464] Registering TS with master...
I20260925 01:16:49.232285 22981 heartbeater.cc:511] Master 127.20.102.190:33339 requested a full tablet report, sending...
I20260925 01:16:49.233532 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.212583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33339 (local address 127.20.102.130:47483)
0925 01:16:49.217312 (+ 4729us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:49.217339 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:49.217530 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:49.217996 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:49.218003 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:49.218647 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:49.219098 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:49.219116 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.220669 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.220703 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:49.223472 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:49.223492 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.224285 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.224294 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:49.224550 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.225381 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:49.225402 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:49.226645 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:49.230457 (+ 3812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:49.230462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:49.230466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:49.231093 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:49.231538 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:49.231543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:49.231547 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:49.231676 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:49.232469 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:49.232475 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:49.232717 (+ 242us) client_negotiation.cc:769] Sending connection context
0925 01:16:49.232859 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 01:16:49.233101 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4512,"thread_start_us":4350,"threads_started":1}
I20260925 01:16:49.233588 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.212628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:47483 (local address 127.20.102.190:33339)
0925 01:16:49.212877 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:49.212886 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:49.217458 (+ 4572us) server_negotiation.cc:407] Connection header received
0925 01:16:49.217734 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:49.217741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:49.217831 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:49.217921 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:49.219270 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.220490 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.223646 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.224072 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.224168 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.226865 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:49.226875 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:49.226880 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:49.226929 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:49.230283 (+ 3354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:49.231227 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:49.231233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:49.231237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:49.231408 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:49.231845 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:49.231851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:49.231857 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:49.232317 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:49.232476 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:49.233011 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 01:16:49.233243 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 01:16:49.233903 22635 ts_manager.cc:194] Registered new tserver with Master: ce342ad910b04a07b732680ce4ab1e08 (127.20.102.130:38731)
I20260925 01:16:49.234905 22635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:34997
I20260925 01:16:49.237051 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 22846
I20260925 01:16:49.237275 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:16:49.237717 20890 external_mini_cluster.cc:1428] Setting key a514758de6d2fe746fda5243f2291fea
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:16:49.279440 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:33339
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:35645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:49.611115 22985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:49.611436 22985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:49.611680 22985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:49.619248 22985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:49.619396 22985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:49.619585 22985 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 01:16:49.619683 22985 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:16:49.619784 22985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:16:49.629631 22985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:35645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.102.190:33339
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:49.631428 22985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:49.633395 22985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:49.647184 22990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:49.647143 22991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:49.647213 22993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:49.648000 22985 server_base.cc:1034] running on GCE node
I20260925 01:16:49.648751 22985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:49.649778 22985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:49.650985 22985 hybrid_clock.cc:648] HybridClock initialized: now 1790299009650937 us; error 42 us; skew 500 ppm
I20260925 01:16:49.651327 22985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:49 dist-test-slave-s12s krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299009, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:49.654776 22985 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:16:49.656343 22985 webserver.cc:460] Webserver started at http://127.20.102.131:39145/ using document root <none> and password file <none>
I20260925 01:16:49.657168 22985 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:49.657312 22985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:49.657636 22985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:49.660457 22985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "3589f87a4d374f3b85f1f8143b3a44c1"
format_stamp: "Formatted at 2026-09-25 01:16:49 on dist-test-slave-s12s"
server_key: "56bd0db6f6a6c51cf93ba3e59f11a707"
server_key_iv: "478aa472f16ee099149875b89bdb9bde"
server_key_version: "encryptionkey@0"
I20260925 01:16:49.661273 22985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "3589f87a4d374f3b85f1f8143b3a44c1"
format_stamp: "Formatted at 2026-09-25 01:16:49 on dist-test-slave-s12s"
server_key: "56bd0db6f6a6c51cf93ba3e59f11a707"
server_key_iv: "478aa472f16ee099149875b89bdb9bde"
server_key_version: "encryptionkey@0"
I20260925 01:16:49.666301 22985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:16:49.669936 23000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:49.671332 22985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:16:49.671566 22985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "3589f87a4d374f3b85f1f8143b3a44c1"
format_stamp: "Formatted at 2026-09-25 01:16:49 on dist-test-slave-s12s"
server_key: "56bd0db6f6a6c51cf93ba3e59f11a707"
server_key_iv: "478aa472f16ee099149875b89bdb9bde"
server_key_version: "encryptionkey@0"
I20260925 01:16:49.671780 22985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:49.714962 22985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:49.719188 22985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:49.719501 22985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:49.722304 22985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:49.722421 22985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:49.722576 22985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:49.722689 22985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:16:49 dist-test-slave-s12s krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299009, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:49.742934 22842 heartbeater.cc:503] Master 127.20.102.190:33339 was elected leader, sending a full tablet report...
I20260925 01:16:49.749325 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.722948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:46007 (local address 127.20.102.190:33339)
0925 01:16:49.723191 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:49.723200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:49.724733 (+ 1533us) server_negotiation.cc:407] Connection header received
0925 01:16:49.727164 (+ 2431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:49.727172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:49.727279 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:49.727430 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:49.730204 (+ 2774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.731383 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.734322 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.734980 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.735119 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.741167 (+ 6048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:49.741182 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:49.741186 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:49.741223 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:49.745600 (+ 4377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:49.746355 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:49.746364 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:49.746370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:49.746582 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:49.747053 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:49.747061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:49.747066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:49.747562 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:49.747733 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:49.748512 (+ 779us) server_negotiation.cc:299] Negotiation successful
0925 01:16:49.748831 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 01:16:49.750211 23009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.723563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33339 (local address 127.20.102.131:46007)
0925 01:16:49.724509 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:49.724572 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:49.726818 (+ 2246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:49.727807 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:49.727820 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:49.728866 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:49.729969 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:49.730005 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.731620 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.731626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:49.734131 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:49.734147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.735456 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.735464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:49.735756 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.736645 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:49.736699 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:49.740915 (+ 4216us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:49.745754 (+ 4839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:49.745761 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:49.745773 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:49.746219 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:49.746703 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:49.746707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:49.746709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:49.746917 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:49.747718 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:49.747724 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:49.748068 (+ 344us) client_negotiation.cc:769] Sending connection context
0925 01:16:49.748299 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 01:16:49.748748 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":364,"threads_started":1}
I20260925 01:16:49.773185 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.757026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:57663 (local address 127.20.102.190:33339)
0925 01:16:49.757278 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:49.757293 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:49.758639 (+ 1346us) server_negotiation.cc:407] Connection header received
0925 01:16:49.758846 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:49.758850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:49.758927 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:49.759015 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:49.760505 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.761521 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.762943 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.763392 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.763558 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.766204 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:49.766214 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:49.766220 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:49.766267 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:49.770133 (+ 3866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:49.770933 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:49.770939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:49.770943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:49.771088 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:49.771482 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:49.771490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:49.771494 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:49.771876 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:49.772047 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:49.772487 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 01:16:49.772763 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 01:16:49.773342 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.757671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33339 (local address 127.20.102.131:57663)
0925 01:16:49.758504 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:49.758535 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:49.758704 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:49.759073 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:49.759081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:49.759863 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:49.760314 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:49.760329 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.761679 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.761686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:49.762784 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:49.762797 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.763598 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.763605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:49.763792 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.764535 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:49.764554 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:49.765923 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:49.770271 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:49.770278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:49.770283 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:49.770794 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:49.771208 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:49.771215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:49.771219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:49.771360 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:49.772007 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:49.772012 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:49.772214 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 01:16:49.772377 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 01:16:49.772789 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":446,"threads_started":1}
I20260925 01:16:49.786547 22985 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:34161
I20260925 01:16:49.786628 23119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:34161 every 8 connection(s)
I20260925 01:16:49.788007 22985 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:16:49.788386 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 22985
I20260925 01:16:49.788600 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:16:49.789247 20890 external_mini_cluster.cc:1428] Setting key 7c97279cdc8cef36d31189cfb53b8d2d
I20260925 01:16:49.790380 23120 heartbeater.cc:347] Connected to a master server at 127.20.102.190:33339
I20260925 01:16:49.790606 23120 heartbeater.cc:464] Registering TS with master...
I20260925 01:16:49.791340 23120 heartbeater.cc:511] Master 127.20.102.190:33339 requested a full tablet report, sending...
I20260925 01:16:49.792748 22635 ts_manager.cc:194] Registered new tserver with Master: 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161)
I20260925 01:16:49.793603 22635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:46007
I20260925 01:16:49.804137 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:16:49 dist-test-slave-s12s krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 01:16:49 dist-test-slave-s12s krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:49.850423 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.836073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.20.102.190:33339)
0925 01:16:49.836324 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:49.836331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:49.836533 (+ 202us) server_negotiation.cc:407] Connection header received
0925 01:16:49.836733 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:49.836738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:49.836806 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:49.836895 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:49.838308 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.839072 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.840262 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.840781 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.840892 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.845040 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:49.845052 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:49.845061 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:49.845096 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:49.847709 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:49.848297 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:49.848301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:49.848303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:49.848420 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:49.848789 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:49.848796 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:49.848799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:49.849174 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:49.849434 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:49.849760 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 01:16:49.850000 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 01:16:49.854887 22635 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:16:49.861756 22635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:16:49.886261 23133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.873890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:34161 (local address 127.0.0.1:41882)
0925 01:16:49.875100 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:49.875157 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:49.875364 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:49.878881 (+ 3517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:49.878885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:49.878907 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:49.879471 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:49.879490 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.881413 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.881418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:49.883423 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:49.883436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.885281 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.885288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:49.885491 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.885559 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 01:16:49.885671 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 01:16:49.885825 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":21,"thread_start_us":767,"threads_started":1}
I20260925 01:16:49.886936 23137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.875709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:38731 (local address 127.0.0.1:48252)
0925 01:16:49.879527 (+ 3818us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:49.879549 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:49.879681 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:49.880310 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:49.880315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:49.880342 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:49.880715 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:49.880726 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.882526 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.882529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:49.884311 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:49.884327 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.886092 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.886100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:49.886280 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.886320 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 01:16:49.886420 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 01:16:49.886548 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3673,"mutex_wait_us":57,"thread_start_us":3495,"threads_started":1}
I20260925 01:16:49.887182 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.874120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41882 (local address 127.20.102.131:34161)
0925 01:16:49.878274 (+ 4154us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:49.878283 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:49.878319 (+ 36us) server_negotiation.cc:407] Connection header received
0925 01:16:49.878410 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:49.878417 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:49.878657 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:49.879463 (+ 806us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:49.879811 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.881253 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.883585 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.885069 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.886409 (+ 1340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.886588 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:49.886657 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 01:16:49.886820 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3967,"thread_start_us":3786,"threads_started":1}
I20260925 01:16:49.887676 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.876239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.20.102.130:38731)
0925 01:16:49.878479 (+ 2240us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:49.878486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:49.879761 (+ 1275us) server_negotiation.cc:407] Connection header received
0925 01:16:49.879876 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:49.879883 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:49.880104 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:49.880308 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:49.880897 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.882366 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.884479 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.885871 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.886055 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.886221 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:49.887183 (+ 962us) server_negotiation.cc:299] Negotiation successful
0925 01:16:49.887368 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2103,"thread_start_us":1961,"threads_started":1}
I20260925 01:16:49.887773 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.874223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:34703 (local address 127.0.0.1:52744)
0925 01:16:49.876468 (+ 2245us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:49.876490 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:49.876649 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:49.878603 (+ 1954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:49.878613 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:49.878645 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:49.879598 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:49.879612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.881434 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.881440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:49.884845 (+ 3405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:49.884860 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.886964 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.886972 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:49.887147 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.887189 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 01:16:49.887283 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 01:16:49.887412 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2090,"mutex_wait_us":446,"thread_start_us":1916,"threads_started":1}
I20260925 01:16:49.888214 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:49.874488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52744 (local address 127.20.102.129:34703)
0925 01:16:49.875977 (+ 1489us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:49.875989 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:49.876817 (+ 828us) server_negotiation.cc:407] Connection header received
0925 01:16:49.876950 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:49.876961 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:49.877278 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:49.877495 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:49.879753 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.881243 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.885029 (+ 3786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:49.886470 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:49.886651 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:49.886797 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:49.887600 (+ 803us) server_negotiation.cc:299] Negotiation successful
0925 01:16:49.887870 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1318,"thread_start_us":1189,"threads_started":1}
I20260925 01:16:49.891026 23050 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf5b583bbd54c4181d7d712923c83a5 (DEFAULT_TABLE table=test-table [id=a3c474a36506462f81d41d52556ffd13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:49.891279 22911 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf5b583bbd54c4181d7d712923c83a5 (DEFAULT_TABLE table=test-table [id=a3c474a36506462f81d41d52556ffd13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:49.891337 22777 tablet_service.cc:1467] Processing CreateTablet for tablet 4bf5b583bbd54c4181d7d712923c83a5 (DEFAULT_TABLE table=test-table [id=a3c474a36506462f81d41d52556ffd13]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:49.892400 23050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf5b583bbd54c4181d7d712923c83a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:49.892705 22777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf5b583bbd54c4181d7d712923c83a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:49.892724 22911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bf5b583bbd54c4181d7d712923c83a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:49.903961 23139 tablet_bootstrap.cc:492] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1: Bootstrap starting.
I20260925 01:16:49.907833 23140 tablet_bootstrap.cc:492] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08: Bootstrap starting.
I20260925 01:16:49.908069 23141 tablet_bootstrap.cc:492] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5: Bootstrap starting.
I20260925 01:16:49.908421 23139 tablet_bootstrap.cc:654] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:49.909842 23139 log.cc:824] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1: Log is configured to *not* fsync() on all Append() calls
I20260925 01:16:49.911605 23141 tablet_bootstrap.cc:654] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:49.911605 23140 tablet_bootstrap.cc:654] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:49.912598 23141 log.cc:824] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5: Log is configured to *not* fsync() on all Append() calls
I20260925 01:16:49.912830 23140 log.cc:824] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08: Log is configured to *not* fsync() on all Append() calls
I20260925 01:16:49.913230 23139 tablet_bootstrap.cc:492] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1: No bootstrap required, opened a new log
I20260925 01:16:49.913610 23139 ts_tablet_manager.cc:1397] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 01:16:49.915418 23140 tablet_bootstrap.cc:492] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08: No bootstrap required, opened a new log
I20260925 01:16:49.915696 23140 ts_tablet_manager.cc:1397] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260925 01:16:49.915745 23141 tablet_bootstrap.cc:492] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5: No bootstrap required, opened a new log
I20260925 01:16:49.915997 23141 ts_tablet_manager.cc:1397] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260925 01:16:49.923132 23140 raft_consensus.cc:348] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } }
I20260925 01:16:49.923511 23140 raft_consensus.cc:374] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:49.923610 23140 raft_consensus.cc:729] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce342ad910b04a07b732680ce4ab1e08, State: Initialized, Role: FOLLOWER
I20260925 01:16:49.924254 23140 consensus_queue.cc:260] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } }
I20260925 01:16:49.924376 23139 raft_consensus.cc:348] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } }
I20260925 01:16:49.924834 23139 raft_consensus.cc:374] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:49.924932 23139 raft_consensus.cc:729] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3589f87a4d374f3b85f1f8143b3a44c1, State: Initialized, Role: FOLLOWER
I20260925 01:16:49.925506 22981 heartbeater.cc:503] Master 127.20.102.190:33339 was elected leader, sending a full tablet report...
I20260925 01:16:49.925516 23139 consensus_queue.cc:260] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } }
I20260925 01:16:49.926316 23140 ts_tablet_manager.cc:1428] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08: Time spent starting tablet: real 0.010s user 0.003s sys 0.007s
I20260925 01:16:49.926682 23141 raft_consensus.cc:348] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } }
I20260925 01:16:49.927210 23141 raft_consensus.cc:374] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:49.927320 23141 raft_consensus.cc:729] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23074871eef248bab79f492dbc710aa5, State: Initialized, Role: FOLLOWER
I20260925 01:16:49.928108 23141 consensus_queue.cc:260] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } }
I20260925 01:16:49.928890 23120 heartbeater.cc:503] Master 127.20.102.190:33339 was elected leader, sending a full tablet report...
I20260925 01:16:49.929509 23139 ts_tablet_manager.cc:1428] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260925 01:16:49.930814 23141 ts_tablet_manager.cc:1428] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260925 01:16:49.981483 22982 tablet.cc:2369] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:16:49.984786 22843 tablet.cc:2369] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:16:50.040084 23121 tablet.cc:2369] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:16:50.116281 23145 raft_consensus.cc:482] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:16:50.116629 23145 raft_consensus.cc:504] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } }
I20260925 01:16:50.118659 23145 leader_election.cc:288] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161), 23074871eef248bab79f492dbc710aa5 (127.20.102.129:34703)
I20260925 01:16:50.121200 23147 raft_consensus.cc:482] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:16:50.121541 23147 raft_consensus.cc:504] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } }
I20260925 01:16:50.127780 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.119146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:48597 (local address 127.20.102.131:34161)
0925 01:16:50.119383 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:50.119390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:50.120275 (+ 885us) server_negotiation.cc:407] Connection header received
0925 01:16:50.120570 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:50.120576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:50.120649 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:50.120751 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:50.121693 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.122740 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.125438 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.126653 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.126776 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.126847 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:50.127382 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 01:16:50.127530 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:16:50.128647 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.119110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:34161 (local address 127.20.102.130:48597)
0925 01:16:50.120159 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:50.120182 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:50.120390 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:50.120941 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:50.120946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:50.121039 (+ 93us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:50.121520 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.121533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.122916 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.122921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.125240 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.125259 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.127030 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.127036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.127205 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.127242 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 01:16:50.128097 (+ 855us) client_negotiation.cc:240] Negotiation successful
0925 01:16:50.128241 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":912,"mutex_wait_us":4,"spinlock_wait_cycles":26368,"thread_start_us":640,"threads_started":1}
I20260925 01:16:50.129153 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.119551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:54261 (local address 127.20.102.129:34703)
0925 01:16:50.119779 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:50.119785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:50.120498 (+ 713us) server_negotiation.cc:407] Connection header received
0925 01:16:50.120723 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:50.120728 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:50.120820 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:50.120910 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:50.121836 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.122960 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.125882 (+ 2922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.127479 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.127632 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.127738 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:50.128632 (+ 894us) server_negotiation.cc:299] Negotiation successful
0925 01:16:50.128833 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 01:16:50.129598 23075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf5b583bbd54c4181d7d712923c83a5" candidate_uuid: "ce342ad910b04a07b732680ce4ab1e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" is_pre_election: true
I20260925 01:16:50.130246 23075 raft_consensus.cc:2454] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce342ad910b04a07b732680ce4ab1e08 in term 0.
I20260925 01:16:50.130378 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.119132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:34703 (local address 127.20.102.130:54261)
0925 01:16:50.120377 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:50.120392 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:50.120552 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:50.120930 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:50.120934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:50.120969 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:50.121632 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.121646 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.123215 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.123220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.125672 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.125691 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.128163 (+ 2472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.128172 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.128410 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.128459 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 01:16:50.129668 (+ 1209us) client_negotiation.cc:240] Negotiation successful
0925 01:16:50.129826 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1126,"mutex_wait_us":29,"thread_start_us":918,"threads_started":1}
I20260925 01:16:50.131024 22862 leader_election.cc:302] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3589f87a4d374f3b85f1f8143b3a44c1, ce342ad910b04a07b732680ce4ab1e08; no voters:
I20260925 01:16:50.131503 23145 raft_consensus.cc:2781] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:16:50.131642 23145 raft_consensus.cc:482] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:16:50.131731 23145 raft_consensus.cc:3037] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:50.131796 22797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf5b583bbd54c4181d7d712923c83a5" candidate_uuid: "ce342ad910b04a07b732680ce4ab1e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23074871eef248bab79f492dbc710aa5" is_pre_election: true
I20260925 01:16:50.132329 22797 raft_consensus.cc:2454] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce342ad910b04a07b732680ce4ab1e08 in term 0.
I20260925 01:16:50.134240 23147 leader_election.cc:288] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161), ce342ad910b04a07b732680ce4ab1e08 (127.20.102.130:38731)
I20260925 01:16:50.135155 23145 raft_consensus.cc:504] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } }
I20260925 01:16:50.136327 22797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf5b583bbd54c4181d7d712923c83a5" candidate_uuid: "ce342ad910b04a07b732680ce4ab1e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23074871eef248bab79f492dbc710aa5"
I20260925 01:16:50.136353 23075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf5b583bbd54c4181d7d712923c83a5" candidate_uuid: "ce342ad910b04a07b732680ce4ab1e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3589f87a4d374f3b85f1f8143b3a44c1"
I20260925 01:16:50.136554 22797 raft_consensus.cc:3037] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:50.136557 23075 raft_consensus.cc:3037] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:50.138033 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.124354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:41867 (local address 127.20.102.131:34161)
0925 01:16:50.127996 (+ 3642us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:50.128002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:50.130201 (+ 2199us) server_negotiation.cc:407] Connection header received
0925 01:16:50.130471 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:50.130475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:50.130549 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:50.130616 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:50.131408 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.132605 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.135416 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.136770 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.136914 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.137002 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:50.137538 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 01:16:50.137697 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3548,"thread_start_us":9707,"threads_started":1}
I20260925 01:16:50.138819 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.123814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:34161 (local address 127.20.102.129:41867)
0925 01:16:50.130107 (+ 6293us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:50.130127 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:50.130315 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:50.130804 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:50.130809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:50.130837 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:50.131240 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.131255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.132804 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.132810 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.135200 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.135217 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.137158 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.137164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.137341 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.137377 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:16:50.138274 (+ 897us) client_negotiation.cc:240] Negotiation successful
0925 01:16:50.138420 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6137,"thread_start_us":5957,"threads_started":1}
I20260925 01:16:50.139525 23074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf5b583bbd54c4181d7d712923c83a5" candidate_uuid: "23074871eef248bab79f492dbc710aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" is_pre_election: true
I20260925 01:16:50.139765 23145 leader_election.cc:288] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [CANDIDATE]: Term 1 election: Requested vote from peers 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161), 23074871eef248bab79f492dbc710aa5 (127.20.102.129:34703)
I20260925 01:16:50.140436 23075 raft_consensus.cc:2454] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce342ad910b04a07b732680ce4ab1e08 in term 1.
I20260925 01:16:50.140839 22797 raft_consensus.cc:2454] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce342ad910b04a07b732680ce4ab1e08 in term 1.
I20260925 01:16:50.140929 22862 leader_election.cc:302] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3589f87a4d374f3b85f1f8143b3a44c1, ce342ad910b04a07b732680ce4ab1e08; no voters:
I20260925 01:16:50.141389 23145 raft_consensus.cc:2781] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:50.141906 23145 raft_consensus.cc:686] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [term 1 LEADER]: Becoming Leader. State: Replica: ce342ad910b04a07b732680ce4ab1e08, State: Running, Role: LEADER
I20260925 01:16:50.142272 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.133927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:43379 (local address 127.20.102.130:38731)
0925 01:16:50.134119 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:50.134125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:50.135094 (+ 969us) server_negotiation.cc:407] Connection header received
0925 01:16:50.135309 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:50.135314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:50.135410 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:50.135520 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:16:50.136234 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.137342 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.139974 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.141231 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.141426 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.141511 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:50.141890 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 01:16:50.142055 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:16:50.142306 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.134278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:38731 (local address 127.20.102.129:43379)
0925 01:16:50.134988 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:50.135004 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:50.135160 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:50.135669 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:50.135674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:50.135699 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:16:50.136046 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.136059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.137530 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.137535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.139794 (+ 2259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.139809 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.141462 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.141472 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.141669 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.141720 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 01:16:50.141844 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 01:16:50.141979 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":365,"threads_started":1}
I20260925 01:16:50.142508 23145 consensus_queue.cc:237] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } }
I20260925 01:16:50.143184 22931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bf5b583bbd54c4181d7d712923c83a5" candidate_uuid: "23074871eef248bab79f492dbc710aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce342ad910b04a07b732680ce4ab1e08" is_pre_election: true
I20260925 01:16:50.144253 22722 leader_election.cc:302] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 23074871eef248bab79f492dbc710aa5; no voters: 3589f87a4d374f3b85f1f8143b3a44c1, ce342ad910b04a07b732680ce4ab1e08
I20260925 01:16:50.144723 23147 raft_consensus.cc:2726] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 01:16:50.148391 22634 catalog_manager.cc:5495] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce342ad910b04a07b732680ce4ab1e08 (127.20.102.130). New cstate: current_term: 1 leader_uuid: "ce342ad910b04a07b732680ce4ab1e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } health_report { overall_health: HEALTHY } } }
Sep 25 01:16:50 dist-test-slave-s12s krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299008, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:50.212581 22729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.199122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40508 (local address 127.20.102.190:33339)
0925 01:16:50.199312 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:50.199317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:50.199861 (+ 544us) server_negotiation.cc:407] Connection header received
0925 01:16:50.200113 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:50.200118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:50.200189 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:50.200310 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:50.201558 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.202299 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.203575 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.204014 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.204171 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.207381 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:50.207386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:50.207390 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:50.207416 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:50.209887 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:50.210573 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:50.210581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:50.210585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:50.210712 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:50.211077 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:50.211084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:50.211095 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:50.211370 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:50.211519 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:50.211925 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 01:16:50.212265 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":74}
I20260925 01:16:50.212654 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.199204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33339 (local address 127.0.0.1:40508)
0925 01:16:50.199759 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:50.199774 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:50.199943 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:50.200331 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:50.200335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:50.201162 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:50.201418 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.201427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.202423 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.202431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.203416 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.203428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.204209 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.204215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.204368 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.205006 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:50.205023 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:50.207218 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:50.210032 (+ 2814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:50.210046 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:50.210053 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:50.210462 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:50.210829 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:50.210836 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:50.210839 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:50.210976 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:50.211494 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:50.211502 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:50.211682 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 01:16:50.211841 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 01:16:50.212131 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":303,"threads_started":1}
I20260925 01:16:50.217749 22634 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:40508:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 01:16:50.218856 22634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:16:50.225639 22777 tablet_service.cc:1467] Processing CreateTablet for tablet c02471e147c44944a1681b223f80e9d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=79b82bf9a2c4400f824fe303d1c01239]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:16:50.225832 22911 tablet_service.cc:1467] Processing CreateTablet for tablet c02471e147c44944a1681b223f80e9d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=79b82bf9a2c4400f824fe303d1c01239]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:16:50.226639 22777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02471e147c44944a1681b223f80e9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:50.226236 23050 tablet_service.cc:1467] Processing CreateTablet for tablet c02471e147c44944a1681b223f80e9d9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=79b82bf9a2c4400f824fe303d1c01239]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:16:50.226818 22911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02471e147c44944a1681b223f80e9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:50.227042 23050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c02471e147c44944a1681b223f80e9d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:50.233846 23141 tablet_bootstrap.cc:492] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5: Bootstrap starting.
I20260925 01:16:50.234159 23139 tablet_bootstrap.cc:492] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1: Bootstrap starting.
I20260925 01:16:50.235399 23141 tablet_bootstrap.cc:654] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:50.235819 23139 tablet_bootstrap.cc:654] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:50.236325 23140 tablet_bootstrap.cc:492] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08: Bootstrap starting.
I20260925 01:16:50.237881 23140 tablet_bootstrap.cc:654] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:50.238554 23141 tablet_bootstrap.cc:492] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5: No bootstrap required, opened a new log
I20260925 01:16:50.238726 23141 ts_tablet_manager.cc:1397] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260925 01:16:50.239976 23141 raft_consensus.cc:348] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.240787 23141 raft_consensus.cc:374] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:50.240985 23141 raft_consensus.cc:729] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23074871eef248bab79f492dbc710aa5, State: Initialized, Role: FOLLOWER
I20260925 01:16:50.241359 23141 consensus_queue.cc:260] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.242151 23141 ts_tablet_manager.cc:1428] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 01:16:50.241973 23147 tablet_replica.cc:439] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } }
I20260925 01:16:50.243029 23147 tablet_replica.cc:442] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:16:50.243830 23139 tablet_bootstrap.cc:492] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1: No bootstrap required, opened a new log
I20260925 01:16:50.244041 23139 ts_tablet_manager.cc:1397] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 01:16:50.244915 23139 raft_consensus.cc:348] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.245254 23139 raft_consensus.cc:374] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:50.245404 23139 raft_consensus.cc:729] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3589f87a4d374f3b85f1f8143b3a44c1, State: Initialized, Role: FOLLOWER
I20260925 01:16:50.245736 23139 consensus_queue.cc:260] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.246532 23139 ts_tablet_manager.cc:1428] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 01:16:50.246441 23146 tablet_replica.cc:439] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } }
I20260925 01:16:50.246995 23146 tablet_replica.cc:442] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:16:50.250391 23140 tablet_bootstrap.cc:492] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08: No bootstrap required, opened a new log
I20260925 01:16:50.250583 23140 ts_tablet_manager.cc:1397] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260925 01:16:50.251381 23140 raft_consensus.cc:348] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.251577 23140 raft_consensus.cc:374] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:50.251724 23140 raft_consensus.cc:729] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce342ad910b04a07b732680ce4ab1e08, State: Initialized, Role: FOLLOWER
I20260925 01:16:50.251962 23140 consensus_queue.cc:260] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.252391 23145 tablet_replica.cc:439] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } }
I20260925 01:16:50.252775 23145 tablet_replica.cc:442] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:16:50.253264 23140 ts_tablet_manager.cc:1428] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 01:16:50.314064 23145 raft_consensus.cc:482] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:16:50.314277 23145 raft_consensus.cc:504] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.314917 23145 leader_election.cc:288] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23074871eef248bab79f492dbc710aa5 (127.20.102.129:34703), 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161)
I20260925 01:16:50.315515 23075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02471e147c44944a1681b223f80e9d9" candidate_uuid: "ce342ad910b04a07b732680ce4ab1e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" is_pre_election: true
I20260925 01:16:50.315511 22797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02471e147c44944a1681b223f80e9d9" candidate_uuid: "ce342ad910b04a07b732680ce4ab1e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23074871eef248bab79f492dbc710aa5" is_pre_election: true
I20260925 01:16:50.315801 23075 raft_consensus.cc:2454] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce342ad910b04a07b732680ce4ab1e08 in term 0.
I20260925 01:16:50.315825 22797 raft_consensus.cc:2454] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce342ad910b04a07b732680ce4ab1e08 in term 0.
I20260925 01:16:50.316267 22862 leader_election.cc:302] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3589f87a4d374f3b85f1f8143b3a44c1, ce342ad910b04a07b732680ce4ab1e08; no voters:
I20260925 01:16:50.316570 23145 raft_consensus.cc:2781] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:16:50.316655 23145 raft_consensus.cc:482] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:16:50.316829 23145 raft_consensus.cc:3037] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:50.318840 23145 raft_consensus.cc:504] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.319355 23145 leader_election.cc:288] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [CANDIDATE]: Term 1 election: Requested vote from peers 23074871eef248bab79f492dbc710aa5 (127.20.102.129:34703), 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161)
I20260925 01:16:50.319814 22797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02471e147c44944a1681b223f80e9d9" candidate_uuid: "ce342ad910b04a07b732680ce4ab1e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23074871eef248bab79f492dbc710aa5"
I20260925 01:16:50.320034 22797 raft_consensus.cc:3037] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:50.320046 23075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c02471e147c44944a1681b223f80e9d9" candidate_uuid: "ce342ad910b04a07b732680ce4ab1e08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3589f87a4d374f3b85f1f8143b3a44c1"
I20260925 01:16:50.320230 23075 raft_consensus.cc:3037] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:50.322011 22797 raft_consensus.cc:2454] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce342ad910b04a07b732680ce4ab1e08 in term 1.
I20260925 01:16:50.322374 22864 leader_election.cc:302] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23074871eef248bab79f492dbc710aa5, ce342ad910b04a07b732680ce4ab1e08; no voters:
I20260925 01:16:50.322363 23075 raft_consensus.cc:2454] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce342ad910b04a07b732680ce4ab1e08 in term 1.
I20260925 01:16:50.322654 23145 raft_consensus.cc:2781] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:50.322836 23145 raft_consensus.cc:686] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [term 1 LEADER]: Becoming Leader. State: Replica: ce342ad910b04a07b732680ce4ab1e08, State: Running, Role: LEADER
I20260925 01:16:50.323104 23145 consensus_queue.cc:237] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.324641 23153 tablet_replica.cc:439] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08: TxnStatusTablet state changed. Reason: New leader ce342ad910b04a07b732680ce4ab1e08. Latest consensus state: current_term: 1 leader_uuid: "ce342ad910b04a07b732680ce4ab1e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } }
I20260925 01:16:50.325019 23153 tablet_replica.cc:442] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:16:50.325997 23165 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 01:16:50.326288 23165 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 01:16:50.326318 22634 catalog_manager.cc:5495] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce342ad910b04a07b732680ce4ab1e08 (127.20.102.130). New cstate: current_term: 1 leader_uuid: "ce342ad910b04a07b732680ce4ab1e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } health_report { overall_health: UNKNOWN } } }
I20260925 01:16:50.346776 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.340301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.20.102.130:38731)
0925 01:16:50.340461 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:50.340465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:50.340754 (+ 289us) server_negotiation.cc:407] Connection header received
0925 01:16:50.341059 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:50.341064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:50.341144 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:50.341266 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:16:50.341676 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.342458 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.344371 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.344952 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.345053 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.345619 (+ 566us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:16:50.346115 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:16:50.346233 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:50.346429 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 01:16:50.346495 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 01:16:50.346853 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.340326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:38731 (local address 127.0.0.1:48258)
0925 01:16:50.340663 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:50.340700 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:50.340837 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:50.341317 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:50.341322 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:50.341336 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:16:50.341527 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.341538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.342595 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.342603 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.344237 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.344253 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.345103 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.345110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.345329 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.345399 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:16:50.346239 (+ 840us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:16:50.346260 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 01:16:50.346354 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 01:16:50.346503 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260925 01:16:50.355727 23075 raft_consensus.cc:1261] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Refusing update from remote peer ce342ad910b04a07b732680ce4ab1e08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:50.355748 22797 raft_consensus.cc:1261] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Refusing update from remote peer ce342ad910b04a07b732680ce4ab1e08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:50.356763 23145 consensus_queue.cc:1035] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:50.357117 23153 consensus_queue.cc:1035] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:50.366912 23147 tablet_replica.cc:439] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5: TxnStatusTablet state changed. Reason: New leader ce342ad910b04a07b732680ce4ab1e08. Latest consensus state: current_term: 1 leader_uuid: "ce342ad910b04a07b732680ce4ab1e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } }
I20260925 01:16:50.367317 23147 tablet_replica.cc:442] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:16:50.367194 23146 tablet_replica.cc:439] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1: TxnStatusTablet state changed. Reason: New leader ce342ad910b04a07b732680ce4ab1e08. Latest consensus state: current_term: 1 leader_uuid: "ce342ad910b04a07b732680ce4ab1e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } }
I20260925 01:16:50.367434 23146 tablet_replica.cc:442] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:16:50.372190 23145 tablet_replica.cc:439] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce342ad910b04a07b732680ce4ab1e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } }
I20260925 01:16:50.372494 23145 tablet_replica.cc:442] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:16:50.372380 23153 tablet_replica.cc:439] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce342ad910b04a07b732680ce4ab1e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } }
I20260925 01:16:50.372606 23153 tablet_replica.cc:442] T c02471e147c44944a1681b223f80e9d9 P ce342ad910b04a07b732680ce4ab1e08: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:16:50.373886 23169 mvcc.cc:205] Tried to move back new op lower bound from 7333064746406170624 to 7333064746285772800. Current Snapshot: MvccSnapshot[applied={T|T < 7333064746406170624}]
I20260925 01:16:50.377486 23170 mvcc.cc:205] Tried to move back new op lower bound from 7333064746406170624 to 7333064746285772800. Current Snapshot: MvccSnapshot[applied={T|T < 7333064746406170624}]
I20260925 01:16:50.384009 23146 tablet_replica.cc:439] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce342ad910b04a07b732680ce4ab1e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } }
I20260925 01:16:50.384325 23146 tablet_replica.cc:442] T c02471e147c44944a1681b223f80e9d9 P 3589f87a4d374f3b85f1f8143b3a44c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:16:50.385702 23147 tablet_replica.cc:439] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce342ad910b04a07b732680ce4ab1e08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } } }
I20260925 01:16:50.385972 23147 tablet_replica.cc:442] T c02471e147c44944a1681b223f80e9d9 P 23074871eef248bab79f492dbc710aa5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:16:50.394811 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.389360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48260 (local address 127.20.102.130:38731)
0925 01:16:50.389624 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:50.389633 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:50.390022 (+ 389us) server_negotiation.cc:407] Connection header received
0925 01:16:50.390235 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:50.390240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:50.390317 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:50.390428 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:16:50.390869 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.391603 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.393089 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.393550 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.393687 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.394094 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:16:50.394192 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:16:50.394300 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:50.394443 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 01:16:50.394536 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 01:16:50.411345 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.405310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:56587 (local address 127.20.102.130:38731)
0925 01:16:50.405542 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:50.405549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:50.406225 (+ 676us) server_negotiation.cc:407] Connection header received
0925 01:16:50.406436 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:50.406441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:50.406512 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:50.406596 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:16:50.407182 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.407924 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.409473 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.409904 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.410064 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.410509 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:16:50.410627 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:16:50.410728 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:50.410864 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 01:16:50.410944 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":53}
I20260925 01:16:50.411407 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.405393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:38731 (local address 127.20.102.130:56587)
0925 01:16:50.406132 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:50.406148 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:50.406323 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:50.406606 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:50.406610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:50.406619 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:16:50.406950 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.406959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.408056 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.408062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.409298 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.409310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.410107 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.410113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.410268 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.410316 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:16:50.410727 (+ 411us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:16:50.410738 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 01:16:50.410836 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 01:16:50.410981 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"mutex_wait_us":4,"thread_start_us":415,"threads_started":1}
I20260925 01:16:50.421051 23075 raft_consensus.cc:1261] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Refusing update from remote peer ce342ad910b04a07b732680ce4ab1e08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:50.421160 22797 raft_consensus.cc:1261] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Refusing update from remote peer ce342ad910b04a07b732680ce4ab1e08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:50.421752 23145 consensus_queue.cc:1035] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:50.422075 23173 consensus_queue.cc:1035] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:50.429993 23168 mvcc.cc:205] Tried to move back new op lower bound from 7333064746679353344 to 7333064745549049856. Current Snapshot: MvccSnapshot[applied={T|T < 7333064746679353344}]
I20260925 01:16:50.491499 22634 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:16:50.491781 22634 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:16:50.495296 22634 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645: Sending DeleteTablet for 3 replicas of tablet 4bf5b583bbd54c4181d7d712923c83a5
I20260925 01:16:50.496382 22777 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bf5b583bbd54c4181d7d712923c83a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:50 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:52744
I20260925 01:16:50.496349 23050 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bf5b583bbd54c4181d7d712923c83a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:50 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:41882
I20260925 01:16:50.497370 23186 tablet_replica.cc:330] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5: stopping tablet replica
I20260925 01:16:50.497560 22911 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bf5b583bbd54c4181d7d712923c83a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:50 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:48252
I20260925 01:16:50.497897 23186 raft_consensus.cc:2229] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:16:50.498315 23186 raft_consensus.cc:2258] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:50.500305 23186 ts_tablet_manager.cc:1905] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:50.500049 22634 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:16:50.500916 22634 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:16:50.508028 23186 ts_tablet_manager.cc:1918] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:16:50.508394 23186 log.cc:1196] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/4bf5b583bbd54c4181d7d712923c83a5
I20260925 01:16:50.508466 23187 tablet_replica.cc:330] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08: stopping tablet replica
I20260925 01:16:50.508927 23187 raft_consensus.cc:2229] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:16:50.509030 23186 ts_tablet_manager.cc:1939] T 4bf5b583bbd54c4181d7d712923c83a5 P 23074871eef248bab79f492dbc710aa5: Deleting consensus metadata
I20260925 01:16:50.509711 23187 raft_consensus.cc:2258] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:50.510509 22911 tablet_service.cc:1467] Processing CreateTablet for tablet 31898b3951214419b938fb8d32b1ad01 (DEFAULT_TABLE table=test-table [id=089115d0983f4a0f938e440c3030fad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:50.510746 22777 tablet_service.cc:1467] Processing CreateTablet for tablet 31898b3951214419b938fb8d32b1ad01 (DEFAULT_TABLE table=test-table [id=089115d0983f4a0f938e440c3030fad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:50.510993 22911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31898b3951214419b938fb8d32b1ad01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:50.510892 23049 tablet_service.cc:1467] Processing CreateTablet for tablet 31898b3951214419b938fb8d32b1ad01 (DEFAULT_TABLE table=test-table [id=089115d0983f4a0f938e440c3030fad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:50.511443 23049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31898b3951214419b938fb8d32b1ad01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:50.511940 23187 ts_tablet_manager.cc:1905] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:50.511140 22777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31898b3951214419b938fb8d32b1ad01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:50.517967 23139 tablet_bootstrap.cc:492] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1: Bootstrap starting.
I20260925 01:16:50.519284 22621 catalog_manager.cc:4841] TS 23074871eef248bab79f492dbc710aa5 (127.20.102.129:34703): tablet 4bf5b583bbd54c4181d7d712923c83a5 (table test-table [id=a3c474a36506462f81d41d52556ffd13]) successfully deleted
I20260925 01:16:50.519613 23139 tablet_bootstrap.cc:654] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:50.519804 23140 tablet_bootstrap.cc:492] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08: Bootstrap starting.
I20260925 01:16:50.520419 23141 tablet_bootstrap.cc:492] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5: Bootstrap starting.
I20260925 01:16:50.522078 23141 tablet_bootstrap.cc:654] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:50.522075 23187 ts_tablet_manager.cc:1918] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:16:50.522378 23187 log.cc:1196] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/4bf5b583bbd54c4181d7d712923c83a5
I20260925 01:16:50.522784 23140 tablet_bootstrap.cc:654] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:50.523188 23187 ts_tablet_manager.cc:1939] T 4bf5b583bbd54c4181d7d712923c83a5 P ce342ad910b04a07b732680ce4ab1e08: Deleting consensus metadata
I20260925 01:16:50.525087 22619 catalog_manager.cc:4841] TS ce342ad910b04a07b732680ce4ab1e08 (127.20.102.130:38731): tablet 4bf5b583bbd54c4181d7d712923c83a5 (table test-table [id=a3c474a36506462f81d41d52556ffd13]) successfully deleted
I20260925 01:16:50.526490 23185 tablet_replica.cc:330] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1: stopping tablet replica
I20260925 01:16:50.527043 23185 raft_consensus.cc:2229] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:16:50.527518 23185 raft_consensus.cc:2258] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:50.528369 23140 tablet_bootstrap.cc:492] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08: No bootstrap required, opened a new log
I20260925 01:16:50.528585 23140 ts_tablet_manager.cc:1397] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20260925 01:16:50.529412 23140 raft_consensus.cc:348] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.529892 23185 ts_tablet_manager.cc:1905] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:50.530362 23140 raft_consensus.cc:374] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:50.530553 23140 raft_consensus.cc:729] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce342ad910b04a07b732680ce4ab1e08, State: Initialized, Role: FOLLOWER
I20260925 01:16:50.530889 23140 consensus_queue.cc:260] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.531581 23140 ts_tablet_manager.cc:1428] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 01:16:50.536895 23139 tablet_bootstrap.cc:492] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1: No bootstrap required, opened a new log
I20260925 01:16:50.537068 23139 ts_tablet_manager.cc:1397] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.000s
I20260925 01:16:50.537781 23139 raft_consensus.cc:348] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.538036 23139 raft_consensus.cc:374] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:50.538143 23139 raft_consensus.cc:729] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3589f87a4d374f3b85f1f8143b3a44c1, State: Initialized, Role: FOLLOWER
I20260925 01:16:50.538476 23139 consensus_queue.cc:260] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.538681 23141 tablet_bootstrap.cc:492] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5: No bootstrap required, opened a new log
I20260925 01:16:50.538854 23141 ts_tablet_manager.cc:1397] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.009s
I20260925 01:16:50.539598 23141 raft_consensus.cc:348] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.539839 23141 raft_consensus.cc:374] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:50.539927 23141 raft_consensus.cc:729] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23074871eef248bab79f492dbc710aa5, State: Initialized, Role: FOLLOWER
I20260925 01:16:50.540046 23139 ts_tablet_manager.cc:1428] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 01:16:50.540180 23141 consensus_queue.cc:260] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.540515 23185 ts_tablet_manager.cc:1918] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:16:50.540733 23185 log.cc:1196] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/4bf5b583bbd54c4181d7d712923c83a5
I20260925 01:16:50.541887 23141 ts_tablet_manager.cc:1428] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 01:16:50.543848 23185 ts_tablet_manager.cc:1939] T 4bf5b583bbd54c4181d7d712923c83a5 P 3589f87a4d374f3b85f1f8143b3a44c1: Deleting consensus metadata
I20260925 01:16:50.545616 22619 catalog_manager.cc:4841] TS 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161): tablet 4bf5b583bbd54c4181d7d712923c83a5 (table test-table [id=a3c474a36506462f81d41d52556ffd13]) successfully deleted
I20260925 01:16:50.573179 23147 raft_consensus.cc:482] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:16:50.573345 23147 raft_consensus.cc:504] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.574011 23147 leader_election.cc:288] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce342ad910b04a07b732680ce4ab1e08 (127.20.102.130:38731), 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161)
I20260925 01:16:50.574572 22931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31898b3951214419b938fb8d32b1ad01" candidate_uuid: "23074871eef248bab79f492dbc710aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce342ad910b04a07b732680ce4ab1e08" is_pre_election: true
I20260925 01:16:50.574720 23075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31898b3951214419b938fb8d32b1ad01" candidate_uuid: "23074871eef248bab79f492dbc710aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" is_pre_election: true
I20260925 01:16:50.574894 22931 raft_consensus.cc:2454] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23074871eef248bab79f492dbc710aa5 in term 0.
I20260925 01:16:50.575008 23075 raft_consensus.cc:2454] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23074871eef248bab79f492dbc710aa5 in term 0.
I20260925 01:16:50.575450 22722 leader_election.cc:302] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23074871eef248bab79f492dbc710aa5, 3589f87a4d374f3b85f1f8143b3a44c1; no voters:
I20260925 01:16:50.575762 23147 raft_consensus.cc:2781] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:16:50.575888 23147 raft_consensus.cc:482] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:16:50.576027 23147 raft_consensus.cc:3037] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:50.578245 23147 raft_consensus.cc:504] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.578873 23147 leader_election.cc:288] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [CANDIDATE]: Term 1 election: Requested vote from peers ce342ad910b04a07b732680ce4ab1e08 (127.20.102.130:38731), 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161)
I20260925 01:16:50.579382 22931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31898b3951214419b938fb8d32b1ad01" candidate_uuid: "23074871eef248bab79f492dbc710aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce342ad910b04a07b732680ce4ab1e08"
I20260925 01:16:50.579638 22931 raft_consensus.cc:3037] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:50.579620 23075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31898b3951214419b938fb8d32b1ad01" candidate_uuid: "23074871eef248bab79f492dbc710aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3589f87a4d374f3b85f1f8143b3a44c1"
I20260925 01:16:50.579819 23075 raft_consensus.cc:3037] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:50.581820 23075 raft_consensus.cc:2454] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23074871eef248bab79f492dbc710aa5 in term 1.
I20260925 01:16:50.582257 22722 leader_election.cc:302] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23074871eef248bab79f492dbc710aa5, 3589f87a4d374f3b85f1f8143b3a44c1; no voters:
I20260925 01:16:50.582520 23147 raft_consensus.cc:2781] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:50.582885 22931 raft_consensus.cc:2454] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23074871eef248bab79f492dbc710aa5 in term 1.
I20260925 01:16:50.583115 23147 raft_consensus.cc:686] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [term 1 LEADER]: Becoming Leader. State: Replica: 23074871eef248bab79f492dbc710aa5, State: Running, Role: LEADER
I20260925 01:16:50.583570 23147 consensus_queue.cc:237] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:50.587095 22634 catalog_manager.cc:5495] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23074871eef248bab79f492dbc710aa5 (127.20.102.129). New cstate: current_term: 1 leader_uuid: "23074871eef248bab79f492dbc710aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } health_report { overall_health: UNKNOWN } } }
I20260925 01:16:50.611038 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.604930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.20.102.129:34703)
0925 01:16:50.605120 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:50.605126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:50.605339 (+ 213us) server_negotiation.cc:407] Connection header received
0925 01:16:50.605504 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:50.605511 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:50.605608 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:50.605764 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:16:50.606052 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.607192 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.608701 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.609295 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.609426 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.609857 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:16:50.610391 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:16:50.610556 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:50.610615 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 01:16:50.610721 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:16:50.627166 23138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.621208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:57643 (local address 127.20.102.130:38731)
0925 01:16:50.621367 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:50.621372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:50.621908 (+ 536us) server_negotiation.cc:407] Connection header received
0925 01:16:50.622105 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:50.622111 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:50.622199 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:50.622297 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:16:50.622776 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.623539 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.624975 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.625441 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.625733 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.626223 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:16:50.626381 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:16:50.626508 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:50.626674 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 01:16:50.626771 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 01:16:50.627215 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.621269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:38731 (local address 127.20.102.129:57643)
0925 01:16:50.621811 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:50.621826 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:50.621978 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:50.622308 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:50.622313 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:50.622324 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:16:50.622611 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.622619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.623676 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.623685 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.624838 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.624851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.625763 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.625770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.625944 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.625998 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:16:50.626523 (+ 525us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:16:50.626537 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 01:16:50.626642 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 01:16:50.626784 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":276,"threads_started":1}
I20260925 01:16:50.639045 23075 raft_consensus.cc:1261] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Refusing update from remote peer 23074871eef248bab79f492dbc710aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:50.639544 22931 raft_consensus.cc:1261] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Refusing update from remote peer 23074871eef248bab79f492dbc710aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:50.639735 23191 consensus_queue.cc:1035] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:50.640211 23147 consensus_queue.cc:1035] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:50.673486 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.667617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:50501 (local address 127.20.102.129:34703)
0925 01:16:50.667777 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:50.667782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:50.667799 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:16:50.667856 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:50.667861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:50.667940 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:50.668036 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:16:50.668429 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.669462 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.671578 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.672046 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.672197 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.672654 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:16:50.672834 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:16:50.672949 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:50.673102 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 01:16:50.673181 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 01:16:50.673605 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:50.666934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:34703 (local address 127.20.102.130:50501)
0925 01:16:50.667270 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:50.667294 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:50.667415 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:50.668062 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:50.668066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:50.668077 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:16:50.668257 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.668265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.669597 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.669603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.671415 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:50.671436 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:50.672218 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:50.672225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:50.672396 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:50.672441 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:16:50.672948 (+ 507us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:16:50.672962 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 01:16:50.673080 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 01:16:50.673227 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196}
W20260925 01:16:50.674990 23169 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 01:16:52.296751 22635 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:16:52.297021 22635 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:16:52.301019 22635 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645: Sending DeleteTablet for 3 replicas of tablet 31898b3951214419b938fb8d32b1ad01
I20260925 01:16:52.301741 22777 tablet_service.cc:1514] Processing DeleteTablet for tablet 31898b3951214419b938fb8d32b1ad01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:52 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:52744
I20260925 01:16:52.301867 22911 tablet_service.cc:1514] Processing DeleteTablet for tablet 31898b3951214419b938fb8d32b1ad01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:52 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:48252
I20260925 01:16:52.301959 23049 tablet_service.cc:1514] Processing DeleteTablet for tablet 31898b3951214419b938fb8d32b1ad01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:52 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:41882
I20260925 01:16:52.314226 23223 tablet_replica.cc:330] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08: stopping tablet replica
I20260925 01:16:52.314543 23223 raft_consensus.cc:2229] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:16:52.314872 23223 raft_consensus.cc:2258] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:52.315611 23222 tablet_replica.cc:330] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5: stopping tablet replica
I20260925 01:16:52.315886 23222 raft_consensus.cc:2229] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:16:52.316533 23222 raft_consensus.cc:2258] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:52.317245 23224 tablet_replica.cc:330] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1: stopping tablet replica
I20260925 01:16:52.317353 23223 ts_tablet_manager.cc:1905] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:52.317441 23224 raft_consensus.cc:2229] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:16:52.317636 23222 ts_tablet_manager.cc:1905] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:52.317682 23224 raft_consensus.cc:2258] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:52.318663 23224 ts_tablet_manager.cc:1905] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:52.324185 23222 ts_tablet_manager.cc:1918] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:16:52.324317 23222 log.cc:1196] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/31898b3951214419b938fb8d32b1ad01
I20260925 01:16:52.324769 23222 ts_tablet_manager.cc:1939] T 31898b3951214419b938fb8d32b1ad01 P 23074871eef248bab79f492dbc710aa5: Deleting consensus metadata
I20260925 01:16:52.324922 23223 ts_tablet_manager.cc:1918] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:16:52.325074 23223 log.cc:1196] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/31898b3951214419b938fb8d32b1ad01
I20260925 01:16:52.325127 23224 ts_tablet_manager.cc:1918] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:16:52.325269 23224 log.cc:1196] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/31898b3951214419b938fb8d32b1ad01
I20260925 01:16:52.325553 23223 ts_tablet_manager.cc:1939] T 31898b3951214419b938fb8d32b1ad01 P ce342ad910b04a07b732680ce4ab1e08: Deleting consensus metadata
I20260925 01:16:52.325698 23224 ts_tablet_manager.cc:1939] T 31898b3951214419b938fb8d32b1ad01 P 3589f87a4d374f3b85f1f8143b3a44c1: Deleting consensus metadata
I20260925 01:16:52.326198 22621 catalog_manager.cc:4841] TS 23074871eef248bab79f492dbc710aa5 (127.20.102.129:34703): tablet 31898b3951214419b938fb8d32b1ad01 (table test-table [id=089115d0983f4a0f938e440c3030fad1]) successfully deleted
I20260925 01:16:52.326769 22619 catalog_manager.cc:4841] TS ce342ad910b04a07b732680ce4ab1e08 (127.20.102.130:38731): tablet 31898b3951214419b938fb8d32b1ad01 (table test-table [id=089115d0983f4a0f938e440c3030fad1]) successfully deleted
I20260925 01:16:52.327014 22619 catalog_manager.cc:4841] TS 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161): tablet 31898b3951214419b938fb8d32b1ad01 (table test-table [id=089115d0983f4a0f938e440c3030fad1]) successfully deleted
Sep 25 01:16:52 dist-test-slave-s12s krb5kdc[22590](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 01:16:52 dist-test-slave-s12s krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:52.361513 23232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:52.347724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.20.102.190:33339)
0925 01:16:52.348241 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:52.348245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:52.348267 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:16:52.348422 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:52.348426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:52.348486 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:52.348581 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:52.349888 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:52.350774 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:52.352010 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:52.352589 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:52.352735 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:52.356555 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:52.356561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:52.356564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:52.356594 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:52.359082 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:52.359722 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:52.359728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:52.359731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:52.359878 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:52.360223 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:52.360226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:52.360228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:52.360503 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:52.360753 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:52.360968 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 01:16:52.361123 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":273,"threads_started":1}
I20260925 01:16:52.364773 22635 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:16:52.365401 22635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:16:52.371641 22777 tablet_service.cc:1467] Processing CreateTablet for tablet a8a3f971bd964764a542f5d4fcf2cb91 (DEFAULT_TABLE table=test-table [id=dfa550d87aa94279bbbb67ff9271a480]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:52.371809 22911 tablet_service.cc:1467] Processing CreateTablet for tablet a8a3f971bd964764a542f5d4fcf2cb91 (DEFAULT_TABLE table=test-table [id=dfa550d87aa94279bbbb67ff9271a480]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:52.372076 22777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a3f971bd964764a542f5d4fcf2cb91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:52.372104 23049 tablet_service.cc:1467] Processing CreateTablet for tablet a8a3f971bd964764a542f5d4fcf2cb91 (DEFAULT_TABLE table=test-table [id=dfa550d87aa94279bbbb67ff9271a480]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:52.372293 22911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a3f971bd964764a542f5d4fcf2cb91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:52.372538 23049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8a3f971bd964764a542f5d4fcf2cb91. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:52.378279 23233 tablet_bootstrap.cc:492] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08: Bootstrap starting.
I20260925 01:16:52.379063 23234 tablet_bootstrap.cc:492] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5: Bootstrap starting.
I20260925 01:16:52.380177 23233 tablet_bootstrap.cc:654] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:52.380795 23234 tablet_bootstrap.cc:654] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:52.382715 23233 tablet_bootstrap.cc:492] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08: No bootstrap required, opened a new log
I20260925 01:16:52.382879 23233 ts_tablet_manager.cc:1397] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 01:16:52.383221 23234 tablet_bootstrap.cc:492] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5: No bootstrap required, opened a new log
I20260925 01:16:52.383410 23234 ts_tablet_manager.cc:1397] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 01:16:52.383679 23233 raft_consensus.cc:348] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.383890 23233 raft_consensus.cc:374] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:52.383989 23233 raft_consensus.cc:729] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce342ad910b04a07b732680ce4ab1e08, State: Initialized, Role: FOLLOWER
I20260925 01:16:52.384225 23235 tablet_bootstrap.cc:492] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1: Bootstrap starting.
I20260925 01:16:52.384246 23233 consensus_queue.cc:260] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.384503 23234 raft_consensus.cc:348] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.384796 23234 raft_consensus.cc:374] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:52.384888 23234 raft_consensus.cc:729] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23074871eef248bab79f492dbc710aa5, State: Initialized, Role: FOLLOWER
I20260925 01:16:52.385169 23234 consensus_queue.cc:260] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.385679 23233 ts_tablet_manager.cc:1428] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 01:16:52.385927 23234 ts_tablet_manager.cc:1428] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 01:16:52.386039 23235 tablet_bootstrap.cc:654] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:52.388335 23235 tablet_bootstrap.cc:492] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1: No bootstrap required, opened a new log
I20260925 01:16:52.388486 23235 ts_tablet_manager.cc:1397] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 01:16:52.389276 23235 raft_consensus.cc:348] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.389532 23235 raft_consensus.cc:374] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:52.389639 23235 raft_consensus.cc:729] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3589f87a4d374f3b85f1f8143b3a44c1, State: Initialized, Role: FOLLOWER
I20260925 01:16:52.389904 23235 consensus_queue.cc:260] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.390698 23235 ts_tablet_manager.cc:1428] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 01:16:52.487759 23212 raft_consensus.cc:482] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:16:52.487938 23212 raft_consensus.cc:504] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.488551 23212 leader_election.cc:288] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce342ad910b04a07b732680ce4ab1e08 (127.20.102.130:38731), 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161)
I20260925 01:16:52.489202 23074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a3f971bd964764a542f5d4fcf2cb91" candidate_uuid: "23074871eef248bab79f492dbc710aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" is_pre_election: true
I20260925 01:16:52.489234 22930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a3f971bd964764a542f5d4fcf2cb91" candidate_uuid: "23074871eef248bab79f492dbc710aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce342ad910b04a07b732680ce4ab1e08" is_pre_election: true
I20260925 01:16:52.489513 22930 raft_consensus.cc:2454] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23074871eef248bab79f492dbc710aa5 in term 0.
I20260925 01:16:52.489565 23074 raft_consensus.cc:2454] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23074871eef248bab79f492dbc710aa5 in term 0.
I20260925 01:16:52.489969 22722 leader_election.cc:302] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23074871eef248bab79f492dbc710aa5, ce342ad910b04a07b732680ce4ab1e08; no voters:
I20260925 01:16:52.490269 23212 raft_consensus.cc:2781] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:16:52.490406 23212 raft_consensus.cc:482] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:16:52.490522 23212 raft_consensus.cc:3037] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:52.492520 23212 raft_consensus.cc:504] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.493044 23212 leader_election.cc:288] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [CANDIDATE]: Term 1 election: Requested vote from peers ce342ad910b04a07b732680ce4ab1e08 (127.20.102.130:38731), 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161)
I20260925 01:16:52.493530 23074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a3f971bd964764a542f5d4fcf2cb91" candidate_uuid: "23074871eef248bab79f492dbc710aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3589f87a4d374f3b85f1f8143b3a44c1"
I20260925 01:16:52.493551 22930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8a3f971bd964764a542f5d4fcf2cb91" candidate_uuid: "23074871eef248bab79f492dbc710aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce342ad910b04a07b732680ce4ab1e08"
I20260925 01:16:52.493746 22930 raft_consensus.cc:3037] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:52.493762 23074 raft_consensus.cc:3037] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:52.496603 23074 raft_consensus.cc:2454] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23074871eef248bab79f492dbc710aa5 in term 1.
I20260925 01:16:52.496623 22930 raft_consensus.cc:2454] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23074871eef248bab79f492dbc710aa5 in term 1.
I20260925 01:16:52.496982 22722 leader_election.cc:302] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23074871eef248bab79f492dbc710aa5, 3589f87a4d374f3b85f1f8143b3a44c1; no voters:
I20260925 01:16:52.497231 23212 raft_consensus.cc:2781] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:52.497370 23212 raft_consensus.cc:686] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [term 1 LEADER]: Becoming Leader. State: Replica: 23074871eef248bab79f492dbc710aa5, State: Running, Role: LEADER
I20260925 01:16:52.497681 23212 consensus_queue.cc:237] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.500162 22634 catalog_manager.cc:5495] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23074871eef248bab79f492dbc710aa5 (127.20.102.129). New cstate: current_term: 1 leader_uuid: "23074871eef248bab79f492dbc710aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } health_report { overall_health: UNKNOWN } } }
I20260925 01:16:52.540313 23242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:52.535642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.20.102.129:34703)
0925 01:16:52.536252 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:52.536256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:52.536274 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:16:52.536321 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:52.536324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:52.536378 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:52.536455 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:16:52.536858 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:52.537593 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:52.538791 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:52.539214 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:52.539303 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:52.539719 (+ 416us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:16:52.539821 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:16:52.539887 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:52.540020 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 01:16:52.540076 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":388,"threads_started":1}
I20260925 01:16:52.549405 23074 raft_consensus.cc:1261] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Refusing update from remote peer 23074871eef248bab79f492dbc710aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:52.549485 22930 raft_consensus.cc:1261] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Refusing update from remote peer 23074871eef248bab79f492dbc710aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:52.550007 23217 consensus_queue.cc:1035] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:52.550341 23212 consensus_queue.cc:1035] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:52.590515 22633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:16:52.590740 22633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:16:52.593493 22633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645: Sending DeleteTablet for 3 replicas of tablet a8a3f971bd964764a542f5d4fcf2cb91
I20260925 01:16:52.594274 22777 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a3f971bd964764a542f5d4fcf2cb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:52 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:52744
I20260925 01:16:52.594457 22911 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a3f971bd964764a542f5d4fcf2cb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:52 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:48252
I20260925 01:16:52.594589 23049 tablet_service.cc:1514] Processing DeleteTablet for tablet a8a3f971bd964764a542f5d4fcf2cb91 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:52 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:41882
I20260925 01:16:52.594774 23222 tablet_replica.cc:330] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5: stopping tablet replica
I20260925 01:16:52.595036 23222 raft_consensus.cc:2229] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:16:52.595103 23223 tablet_replica.cc:330] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08: stopping tablet replica
I20260925 01:16:52.595328 23223 raft_consensus.cc:2229] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:16:52.595381 23222 raft_consensus.cc:2258] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:52.595575 23223 raft_consensus.cc:2258] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:52.595651 23224 tablet_replica.cc:330] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1: stopping tablet replica
I20260925 01:16:52.595561 22633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48920:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 01:16:52.595834 23224 raft_consensus.cc:2229] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:16:52.596032 23224 raft_consensus.cc:2258] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 01:16:52.596300 22633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:16:52.597004 23224 ts_tablet_manager.cc:1905] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:52.597226 23222 ts_tablet_manager.cc:1905] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:52.602522 23222 ts_tablet_manager.cc:1918] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:16:52.602674 23222 log.cc:1196] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/a8a3f971bd964764a542f5d4fcf2cb91
I20260925 01:16:52.603278 23224 ts_tablet_manager.cc:1918] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:16:52.603404 23222 ts_tablet_manager.cc:1939] T a8a3f971bd964764a542f5d4fcf2cb91 P 23074871eef248bab79f492dbc710aa5: Deleting consensus metadata
I20260925 01:16:52.603328 22777 tablet_service.cc:1467] Processing CreateTablet for tablet 324427eec18a41eaae8ca563a532beb5 (DEFAULT_TABLE table=test-table [id=79fab11223a2499c99cfcf4e3e76aaad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:52.603475 23224 log.cc:1196] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/a8a3f971bd964764a542f5d4fcf2cb91
I20260925 01:16:52.603776 22777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324427eec18a41eaae8ca563a532beb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:52.603868 23224 ts_tablet_manager.cc:1939] T a8a3f971bd964764a542f5d4fcf2cb91 P 3589f87a4d374f3b85f1f8143b3a44c1: Deleting consensus metadata
I20260925 01:16:52.604465 22621 catalog_manager.cc:4841] TS 23074871eef248bab79f492dbc710aa5 (127.20.102.129:34703): tablet a8a3f971bd964764a542f5d4fcf2cb91 (table test-table [id=dfa550d87aa94279bbbb67ff9271a480]) successfully deleted
I20260925 01:16:52.604581 22911 tablet_service.cc:1467] Processing CreateTablet for tablet 324427eec18a41eaae8ca563a532beb5 (DEFAULT_TABLE table=test-table [id=79fab11223a2499c99cfcf4e3e76aaad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:52.605028 22911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324427eec18a41eaae8ca563a532beb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:52.605288 23223 ts_tablet_manager.cc:1905] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:52.607545 22619 catalog_manager.cc:4841] TS 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161): tablet a8a3f971bd964764a542f5d4fcf2cb91 (table test-table [id=dfa550d87aa94279bbbb67ff9271a480]) successfully deleted
I20260925 01:16:52.607970 23049 tablet_service.cc:1467] Processing CreateTablet for tablet 324427eec18a41eaae8ca563a532beb5 (DEFAULT_TABLE table=test-table [id=79fab11223a2499c99cfcf4e3e76aaad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:16:52.608341 23049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 324427eec18a41eaae8ca563a532beb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:52.610276 23223 ts_tablet_manager.cc:1918] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:16:52.610458 23223 log.cc:1196] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/a8a3f971bd964764a542f5d4fcf2cb91
I20260925 01:16:52.610864 23223 ts_tablet_manager.cc:1939] T a8a3f971bd964764a542f5d4fcf2cb91 P ce342ad910b04a07b732680ce4ab1e08: Deleting consensus metadata
I20260925 01:16:52.611765 22619 catalog_manager.cc:4841] TS ce342ad910b04a07b732680ce4ab1e08 (127.20.102.130:38731): tablet a8a3f971bd964764a542f5d4fcf2cb91 (table test-table [id=dfa550d87aa94279bbbb67ff9271a480]) successfully deleted
I20260925 01:16:52.612119 23233 tablet_bootstrap.cc:492] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08: Bootstrap starting.
I20260925 01:16:52.613363 23233 tablet_bootstrap.cc:654] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:52.614220 23234 tablet_bootstrap.cc:492] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5: Bootstrap starting.
I20260925 01:16:52.614606 23235 tablet_bootstrap.cc:492] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1: Bootstrap starting.
I20260925 01:16:52.615821 23234 tablet_bootstrap.cc:654] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:52.615904 23233 tablet_bootstrap.cc:492] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08: No bootstrap required, opened a new log
I20260925 01:16:52.616073 23233 ts_tablet_manager.cc:1397] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 01:16:52.616181 23235 tablet_bootstrap.cc:654] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:52.616952 23233 raft_consensus.cc:348] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.617252 23233 raft_consensus.cc:374] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:52.617357 23233 raft_consensus.cc:729] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce342ad910b04a07b732680ce4ab1e08, State: Initialized, Role: FOLLOWER
I20260925 01:16:52.617650 23233 consensus_queue.cc:260] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.618839 23233 ts_tablet_manager.cc:1428] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 01:16:52.618884 23234 tablet_bootstrap.cc:492] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5: No bootstrap required, opened a new log
I20260925 01:16:52.619030 23234 ts_tablet_manager.cc:1397] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 01:16:52.619047 23235 tablet_bootstrap.cc:492] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1: No bootstrap required, opened a new log
I20260925 01:16:52.619184 23235 ts_tablet_manager.cc:1397] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 01:16:52.619683 23235 raft_consensus.cc:348] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.619828 23234 raft_consensus.cc:348] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.620080 23234 raft_consensus.cc:374] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:52.620179 23234 raft_consensus.cc:729] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23074871eef248bab79f492dbc710aa5, State: Initialized, Role: FOLLOWER
I20260925 01:16:52.620434 23234 consensus_queue.cc:260] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.620579 23235 raft_consensus.cc:374] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:52.621001 23234 ts_tablet_manager.cc:1428] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 01:16:52.621409 23235 raft_consensus.cc:729] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3589f87a4d374f3b85f1f8143b3a44c1, State: Initialized, Role: FOLLOWER
I20260925 01:16:52.621680 23235 consensus_queue.cc:260] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.622181 23235 ts_tablet_manager.cc:1428] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260925 01:16:52.672711 23212 raft_consensus.cc:482] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:16:52.672904 23212 raft_consensus.cc:504] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.673534 23212 leader_election.cc:288] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce342ad910b04a07b732680ce4ab1e08 (127.20.102.130:38731), 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161)
I20260925 01:16:52.674189 22930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324427eec18a41eaae8ca563a532beb5" candidate_uuid: "23074871eef248bab79f492dbc710aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce342ad910b04a07b732680ce4ab1e08" is_pre_election: true
I20260925 01:16:52.674273 23075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324427eec18a41eaae8ca563a532beb5" candidate_uuid: "23074871eef248bab79f492dbc710aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" is_pre_election: true
I20260925 01:16:52.674468 22930 raft_consensus.cc:2454] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23074871eef248bab79f492dbc710aa5 in term 0.
I20260925 01:16:52.674487 23075 raft_consensus.cc:2454] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23074871eef248bab79f492dbc710aa5 in term 0.
I20260925 01:16:52.674882 22722 leader_election.cc:302] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23074871eef248bab79f492dbc710aa5, ce342ad910b04a07b732680ce4ab1e08; no voters:
I20260925 01:16:52.675143 23212 raft_consensus.cc:2781] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:16:52.675297 23212 raft_consensus.cc:482] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:16:52.675432 23212 raft_consensus.cc:3037] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:52.677529 23212 raft_consensus.cc:504] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.678074 23212 leader_election.cc:288] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [CANDIDATE]: Term 1 election: Requested vote from peers ce342ad910b04a07b732680ce4ab1e08 (127.20.102.130:38731), 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161)
I20260925 01:16:52.678529 22930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324427eec18a41eaae8ca563a532beb5" candidate_uuid: "23074871eef248bab79f492dbc710aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce342ad910b04a07b732680ce4ab1e08"
I20260925 01:16:52.678722 22930 raft_consensus.cc:3037] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:52.678623 23075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "324427eec18a41eaae8ca563a532beb5" candidate_uuid: "23074871eef248bab79f492dbc710aa5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3589f87a4d374f3b85f1f8143b3a44c1"
I20260925 01:16:52.678884 23075 raft_consensus.cc:3037] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:52.681218 22930 raft_consensus.cc:2454] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23074871eef248bab79f492dbc710aa5 in term 1.
I20260925 01:16:52.681473 23075 raft_consensus.cc:2454] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23074871eef248bab79f492dbc710aa5 in term 1.
I20260925 01:16:52.681643 22722 leader_election.cc:302] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23074871eef248bab79f492dbc710aa5, ce342ad910b04a07b732680ce4ab1e08; no voters:
I20260925 01:16:52.681963 23212 raft_consensus.cc:2781] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:52.682124 23212 raft_consensus.cc:686] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [term 1 LEADER]: Becoming Leader. State: Replica: 23074871eef248bab79f492dbc710aa5, State: Running, Role: LEADER
I20260925 01:16:52.682399 23212 consensus_queue.cc:237] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } }
I20260925 01:16:52.684818 22635 catalog_manager.cc:5495] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23074871eef248bab79f492dbc710aa5 (127.20.102.129). New cstate: current_term: 1 leader_uuid: "23074871eef248bab79f492dbc710aa5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23074871eef248bab79f492dbc710aa5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 } health_report { overall_health: UNKNOWN } } }
I20260925 01:16:52.714782 23075 raft_consensus.cc:1261] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Refusing update from remote peer 23074871eef248bab79f492dbc710aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:52.715049 22930 raft_consensus.cc:1261] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Refusing update from remote peer 23074871eef248bab79f492dbc710aa5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:16:52.715360 23212 consensus_queue.cc:1035] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3589f87a4d374f3b85f1f8143b3a44c1" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 34161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:16:52.716037 23212 consensus_queue.cc:1035] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce342ad910b04a07b732680ce4ab1e08" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:16:52.733523 23205 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 01:16:54.312217 22635 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:16:54.312466 22635 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48920:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:16:54.316394 22635 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5aad964d4d540818b37d547fa641645: Sending DeleteTablet for 3 replicas of tablet 324427eec18a41eaae8ca563a532beb5
I20260925 01:16:54.317107 22777 tablet_service.cc:1514] Processing DeleteTablet for tablet 324427eec18a41eaae8ca563a532beb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:54 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:52744
I20260925 01:16:54.317231 22911 tablet_service.cc:1514] Processing DeleteTablet for tablet 324427eec18a41eaae8ca563a532beb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:54 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:48252
I20260925 01:16:54.317427 23049 tablet_service.cc:1514] Processing DeleteTablet for tablet 324427eec18a41eaae8ca563a532beb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:16:54 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:41882
I20260925 01:16:54.318063 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 22706
I20260925 01:16:54.332644 23277 tablet_replica.cc:330] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08: stopping tablet replica
I20260925 01:16:54.333231 23278 tablet_replica.cc:330] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1: stopping tablet replica
I20260925 01:16:54.333580 23278 raft_consensus.cc:2229] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:16:54.333811 23276 tablet_replica.cc:330] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5: stopping tablet replica
I20260925 01:16:54.333818 23277 raft_consensus.cc:2229] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:16:54.333963 23278 raft_consensus.cc:2258] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:54.334043 23276 raft_consensus.cc:2229] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:16:54.334074 23277 raft_consensus.cc:2258] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:54.334434 23276 raft_consensus.cc:2258] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:16:54.335204 23277 ts_tablet_manager.cc:1905] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:54.335443 23276 ts_tablet_manager.cc:1905] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:54.335942 23278 ts_tablet_manager.cc:1905] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:16:54.341346 23278 ts_tablet_manager.cc:1918] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:16:54.341503 23278 log.cc:1196] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/324427eec18a41eaae8ca563a532beb5
I20260925 01:16:54.341928 23278 ts_tablet_manager.cc:1939] T 324427eec18a41eaae8ca563a532beb5 P 3589f87a4d374f3b85f1f8143b3a44c1: Deleting consensus metadata
I20260925 01:16:54.342213 23276 ts_tablet_manager.cc:1918] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:16:54.342383 23276 log.cc:1196] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/324427eec18a41eaae8ca563a532beb5
I20260925 01:16:54.342392 23277 ts_tablet_manager.cc:1918] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 01:16:54.342536 23277 log.cc:1196] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/324427eec18a41eaae8ca563a532beb5
I20260925 01:16:54.342859 23276 ts_tablet_manager.cc:1939] T 324427eec18a41eaae8ca563a532beb5 P 23074871eef248bab79f492dbc710aa5: Deleting consensus metadata
I20260925 01:16:54.342939 23277 ts_tablet_manager.cc:1939] T 324427eec18a41eaae8ca563a532beb5 P ce342ad910b04a07b732680ce4ab1e08: Deleting consensus metadata
I20260925 01:16:54.343154 22619 catalog_manager.cc:4841] TS 3589f87a4d374f3b85f1f8143b3a44c1 (127.20.102.131:34161): tablet 324427eec18a41eaae8ca563a532beb5 (table test-table [id=79fab11223a2499c99cfcf4e3e76aaad]) successfully deleted
I20260925 01:16:54.343906 22621 catalog_manager.cc:4841] TS 23074871eef248bab79f492dbc710aa5 (127.20.102.129:34703): tablet 324427eec18a41eaae8ca563a532beb5 (table test-table [id=79fab11223a2499c99cfcf4e3e76aaad]) successfully deleted
I20260925 01:16:54.344064 22619 catalog_manager.cc:4841] TS ce342ad910b04a07b732680ce4ab1e08 (127.20.102.130:38731): tablet 324427eec18a41eaae8ca563a532beb5 (table test-table [id=79fab11223a2499c99cfcf4e3e76aaad]) successfully deleted
Sep 25 01:16:54 dist-test-slave-s12s krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:16:54.352097 23280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:54.320748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34908 (local address 127.20.102.129:34703)
0925 01:16:54.330815 (+ 10067us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:54.330824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:54.330853 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:16:54.330924 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:54.330932 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:54.331067 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:54.333183 (+ 2116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:54.333347 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:54.334657 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:54.336520 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:54.337235 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:54.337955 (+ 720us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:54.346499 (+ 8544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:54.346517 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:54.346535 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:54.346566 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:54.349512 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:54.350169 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:54.350177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:54.350180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:54.350350 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:54.350679 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:54.350682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:54.350684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:54.351120 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:54.351329 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:54.351570 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 01:16:54.351766 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9897,"thread_start_us":9773,"threads_started":1}
W20260925 01:16:54.352759 22837 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34908
W20260925 01:16:54.353479 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 22706: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:16:54.353729 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 22706
I20260925 01:16:54.378213 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 22846
Sep 25 01:16:54 dist-test-slave-s12s krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:16:54.392597 23281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:54.378918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.20.102.130:38731)
0925 01:16:54.379468 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:54.379472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:54.379488 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:16:54.379550 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:54.379553 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:54.379610 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:54.379704 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:54.380891 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:54.381625 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:54.382871 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:54.383265 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:54.383357 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:54.387242 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:54.387255 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:54.387270 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:54.387299 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:54.390270 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:54.390871 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:54.390877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:54.390879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:54.390996 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:54.391299 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:54.391302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:54.391304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:54.391633 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:54.391872 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:54.392056 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 01:16:54.392208 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":312,"threads_started":1}
W20260925 01:16:54.393285 22976 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52466
W20260925 01:16:54.393936 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 22846: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:16:54.394073 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 22846
I20260925 01:16:54.419536 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 22985
Sep 25 01:16:54 dist-test-slave-s12s krb5kdc[22590](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299012, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:16:54.434998 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:54.420331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44696 (local address 127.20.102.131:34161)
0925 01:16:54.420939 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:54.420944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:54.420960 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:16:54.421022 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:54.421025 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:54.421092 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:54.421207 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:54.422491 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:54.423256 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:54.424501 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:54.425071 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:54.425230 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:54.429415 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:54.429431 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:54.429448 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:54.429487 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:54.432204 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:54.432948 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:54.432956 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:54.432958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:54.433122 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:54.433496 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:54.433502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:54.433505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:54.433859 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:54.434166 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:54.434450 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 01:16:54.434675 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":357,"threads_started":1}
W20260925 01:16:54.435714 23115 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44696
W20260925 01:16:54.436419 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 22985: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:16:54.436560 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 22985
I20260925 01:16:54.454856 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 22603
I20260925 01:16:54.456079 22675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:16:54.570827 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 22603
2026-09-25T01:16:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8168 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:16:54 dist-test-slave-s12s krb5kdc[23286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:16:54 dist-test-slave-s12s krb5kdc[23286](info): set up 2 sockets
Sep 25 01:16:54 dist-test-slave-s12s krb5kdc[23286](info): commencing operation
krb5kdc: starting...
W20260925 01:16:55.653193 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:16:55 dist-test-slave-s12s krb5kdc[23286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:16:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:16:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:16:55.835938 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:45537
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:45537
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:56.157917 23299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:56.158209 23299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:56.158377 23299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:56.165546 23299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:16:56.165638 23299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:56.165737 23299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:56.165838 23299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:16:56.165917 23299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:16:56.175455 23299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:45537
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:45537
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:56.176661 23299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:56.177963 23299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:56.189863 23304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:56.189805 23305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:56.189841 23307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:56.190567 23299 server_base.cc:1034] running on GCE node
I20260925 01:16:56.191323 23299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:56.192739 23299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:56.193975 23299 hybrid_clock.cc:648] HybridClock initialized: now 1790299016193927 us; error 53 us; skew 500 ppm
I20260925 01:16:56.194315 23299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:56 dist-test-slave-s12s krb5kdc[23286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299016, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:56.197330 23299 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:16:56.198659 23299 webserver.cc:460] Webserver started at http://127.20.102.190:40993/ using document root <none> and password file <none>
I20260925 01:16:56.199260 23299 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:56.199355 23299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:56.199597 23299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:56.202392 23299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "93df3a73f42d40b89f719e57b06e99bd"
format_stamp: "Formatted at 2026-09-25 01:16:56 on dist-test-slave-s12s"
server_key: "92ff64afff960d72caff189a9e3fba22"
server_key_iv: "b2645281a65c83a24d89751c848873b5"
server_key_version: "encryptionkey@0"
I20260925 01:16:56.203089 23299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "93df3a73f42d40b89f719e57b06e99bd"
format_stamp: "Formatted at 2026-09-25 01:16:56 on dist-test-slave-s12s"
server_key: "92ff64afff960d72caff189a9e3fba22"
server_key_iv: "b2645281a65c83a24d89751c848873b5"
server_key_version: "encryptionkey@0"
I20260925 01:16:56.208487 23299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:16:56.211817 23314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:56.213220 23299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:16:56.213394 23299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "93df3a73f42d40b89f719e57b06e99bd"
format_stamp: "Formatted at 2026-09-25 01:16:56 on dist-test-slave-s12s"
server_key: "92ff64afff960d72caff189a9e3fba22"
server_key_iv: "b2645281a65c83a24d89751c848873b5"
server_key_version: "encryptionkey@0"
I20260925 01:16:56.213574 23299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:56.226024 23299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:56.229784 23299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:56.230063 23299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:56.248433 23299 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:45537
I20260925 01:16:56.248451 23365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:45537 every 8 connection(s)
I20260925 01:16:56.249792 23299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:16:56.251971 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 23299
I20260925 01:16:56.252180 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:16:56.252894 20890 external_mini_cluster.cc:1428] Setting key b8d54e85d5bc2758e0d532b0b4159008
I20260925 01:16:56.253719 23366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:56.267658 23366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd: Bootstrap starting.
Sep 25 01:16:56 dist-test-slave-s12s krb5kdc[23286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:56.271903 23366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:56.273360 23366 log.cc:824] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd: Log is configured to *not* fsync() on all Append() calls
I20260925 01:16:56.276834 23366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd: No bootstrap required, opened a new log
I20260925 01:16:56.278105 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:56.254892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.20.102.190:45537)
0925 01:16:56.255697 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:56.255712 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:56.255764 (+ 52us) server_negotiation.cc:407] Connection header received
0925 01:16:56.258307 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:56.258334 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:56.258831 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:56.259246 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:56.260695 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:56.262629 (+ 1934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:56.263868 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:56.265229 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:56.265457 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:56.269254 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:56.269266 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:56.269293 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:56.269357 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:56.273614 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:56.274475 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:56.274482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:56.274490 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:56.274658 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:56.275055 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:56.275060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:56.275062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:56.275740 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:56.276036 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:56.276393 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 01:16:56.276814 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":331,"threads_started":1}
I20260925 01:16:56.285218 23366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93df3a73f42d40b89f719e57b06e99bd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45537 } }
I20260925 01:16:56.285534 23366 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:56.285606 23366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93df3a73f42d40b89f719e57b06e99bd, State: Initialized, Role: FOLLOWER
I20260925 01:16:56.286109 23366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93df3a73f42d40b89f719e57b06e99bd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45537 } }
I20260925 01:16:56.286302 23366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:16:56.286396 23366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:16:56.286525 23366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:56.288465 23366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93df3a73f42d40b89f719e57b06e99bd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45537 } }
I20260925 01:16:56.288954 23366 leader_election.cc:302] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93df3a73f42d40b89f719e57b06e99bd; no voters:
I20260925 01:16:56.289449 23366 leader_election.cc:288] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:16:56.289546 23371 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:56.290230 23371 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [term 1 LEADER]: Becoming Leader. State: Replica: 93df3a73f42d40b89f719e57b06e99bd, State: Running, Role: LEADER
I20260925 01:16:56.290843 23371 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93df3a73f42d40b89f719e57b06e99bd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45537 } }
I20260925 01:16:56.291664 23366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:16:56.293645 23373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93df3a73f42d40b89f719e57b06e99bd. Latest consensus state: current_term: 1 leader_uuid: "93df3a73f42d40b89f719e57b06e99bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93df3a73f42d40b89f719e57b06e99bd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45537 } } }
I20260925 01:16:56.294080 23373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [sys.catalog]: This master's current role is: LEADER
I20260925 01:16:56.294876 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93df3a73f42d40b89f719e57b06e99bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93df3a73f42d40b89f719e57b06e99bd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45537 } } }
I20260925 01:16:56.295363 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd [sys.catalog]: This master's current role is: LEADER
I20260925 01:16:56.295594 23379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:16:56.300192 23379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:16:56.308456 23379 catalog_manager.cc:1347] Generated new cluster ID: 12698f6508fb443fb4206e3be20d833c
I20260925 01:16:56.308583 23379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:16:56.350032 23379 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:16:56.351958 23379 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:16:56.368249 23379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93df3a73f42d40b89f719e57b06e99bd: Generated new TSK 0
I20260925 01:16:56.369310 23379 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:16:56.428452 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:45537
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:56.751704 23392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:56.751997 23392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:56.752163 23392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:56.759061 23392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:56.759193 23392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:56.759364 23392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:16:56.769018 23392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:45537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:56.770249 23392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:56.771546 23392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:56.785094 23397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:56.785058 23400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:56.785053 23398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:56.785686 23392 server_base.cc:1034] running on GCE node
I20260925 01:16:56.786258 23392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:56.787122 23392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:56.788347 23392 hybrid_clock.cc:648] HybridClock initialized: now 1790299016788298 us; error 59 us; skew 500 ppm
I20260925 01:16:56.788656 23392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:56 dist-test-slave-s12s krb5kdc[23286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299016, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:56.791711 23392 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:16:56.793182 23392 webserver.cc:460] Webserver started at http://127.20.102.129:45539/ using document root <none> and password file <none>
I20260925 01:16:56.793835 23392 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:56.793987 23392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:56.794291 23392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:56.797323 23392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "4977a4f1c7de40518ddf6a886e4e44ec"
format_stamp: "Formatted at 2026-09-25 01:16:56 on dist-test-slave-s12s"
server_key: "8568ed950632dcc29f9375048881b6f0"
server_key_iv: "549c300047a4d9fc02ea43b13176c5b4"
server_key_version: "encryptionkey@0"
I20260925 01:16:56.798139 23392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "4977a4f1c7de40518ddf6a886e4e44ec"
format_stamp: "Formatted at 2026-09-25 01:16:56 on dist-test-slave-s12s"
server_key: "8568ed950632dcc29f9375048881b6f0"
server_key_iv: "549c300047a4d9fc02ea43b13176c5b4"
server_key_version: "encryptionkey@0"
I20260925 01:16:56.803026 23392 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:16:56.806318 23407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:56.807592 23392 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:16:56.807765 23392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "4977a4f1c7de40518ddf6a886e4e44ec"
format_stamp: "Formatted at 2026-09-25 01:16:56 on dist-test-slave-s12s"
server_key: "8568ed950632dcc29f9375048881b6f0"
server_key_iv: "549c300047a4d9fc02ea43b13176c5b4"
server_key_version: "encryptionkey@0"
I20260925 01:16:56.807963 23392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:56.820142 23392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:56.823978 23392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:56.824290 23392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:56.825217 23392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:16:56.826593 23392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:56.826694 23392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:56.826849 23392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:56.826925 23392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:56.859320 23392 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:44065
I20260925 01:16:56.859359 23519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:44065 every 8 connection(s)
I20260925 01:16:56.860643 23392 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:16:56.866947 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 23392
I20260925 01:16:56.867175 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:16:56.868240 20890 external_mini_cluster.cc:1428] Setting key af42c7bf2c18f6e8b5b95f2ea2ab9cda
Sep 25 01:16:56 dist-test-slave-s12s krb5kdc[23286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299016, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:56.881556 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:56.863113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:36919 (local address 127.20.102.190:45537)
0925 01:16:56.863555 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:56.863559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:56.864391 (+ 832us) server_negotiation.cc:407] Connection header received
0925 01:16:56.866276 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:56.866282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:56.866375 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:56.866497 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:56.868260 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:56.869329 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:56.871057 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:56.871470 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:56.871607 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:56.875185 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:56.875190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:56.875194 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:56.875224 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:56.878271 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:56.878969 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:56.878978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:56.878984 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:56.879132 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:56.879524 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:56.879532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:56.879544 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:56.879916 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:56.880060 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:56.880851 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 01:16:56.881090 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":231,"threads_started":1}
I20260925 01:16:56.882246 23523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:56.863539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:45537 (local address 127.20.102.129:36919)
0925 01:16:56.864223 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:56.864267 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:56.866003 (+ 1736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:56.866740 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:56.866749 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:56.867482 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:56.868089 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:56.868106 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:56.869478 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:56.869485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:56.870875 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:56.870887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:56.871692 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:56.871699 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:56.871919 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:56.872611 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:56.872634 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:56.874964 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:56.878430 (+ 3466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:56.878436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:56.878447 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:56.878844 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:56.879243 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:56.879246 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:56.879248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:56.879417 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:56.880065 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:56.880071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:56.880449 (+ 378us) client_negotiation.cc:769] Sending connection context
0925 01:16:56.880755 (+ 306us) client_negotiation.cc:240] Negotiation successful
0925 01:16:56.881158 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":268,"threads_started":1}
I20260925 01:16:56.883787 23520 heartbeater.cc:347] Connected to a master server at 127.20.102.190:45537
I20260925 01:16:56.884135 23520 heartbeater.cc:464] Registering TS with master...
I20260925 01:16:56.885103 23520 heartbeater.cc:511] Master 127.20.102.190:45537 requested a full tablet report, sending...
I20260925 01:16:56.887301 23331 ts_manager.cc:194] Registered new tserver with Master: 4977a4f1c7de40518ddf6a886e4e44ec (127.20.102.129:44065)
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
I20260925 01:16:56.888820 23331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:36919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:16:56.912506 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:45537
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:57.243924 23526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:57.244184 23526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:57.244273 23526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:57.251479 23526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:57.251580 23526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:57.251757 23526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:16:57.261082 23526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:45537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:57.262271 23526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:57.263545 23526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:57.276872 23532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:57.276945 23531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:57.277010 23534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:57.277438 23526 server_base.cc:1034] running on GCE node
I20260925 01:16:57.278060 23526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:57.278963 23526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:57.280201 23526 hybrid_clock.cc:648] HybridClock initialized: now 1790299017280150 us; error 39 us; skew 500 ppm
I20260925 01:16:57.280529 23526 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:57 dist-test-slave-s12s krb5kdc[23286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:57.283562 23526 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:16:57.284947 23526 webserver.cc:460] Webserver started at http://127.20.102.130:35837/ using document root <none> and password file <none>
I20260925 01:16:57.285599 23526 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:57.285729 23526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:57.286005 23526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:57.288808 23526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "f9a0fa48d31346f99dbfe32700c077e0"
format_stamp: "Formatted at 2026-09-25 01:16:57 on dist-test-slave-s12s"
server_key: "a5f1fc80443e8a560fcd2a87f4d022d2"
server_key_iv: "bbab97dfe33b2fa9b2693764d4cef7db"
server_key_version: "encryptionkey@0"
I20260925 01:16:57.289575 23526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "f9a0fa48d31346f99dbfe32700c077e0"
format_stamp: "Formatted at 2026-09-25 01:16:57 on dist-test-slave-s12s"
server_key: "a5f1fc80443e8a560fcd2a87f4d022d2"
server_key_iv: "bbab97dfe33b2fa9b2693764d4cef7db"
server_key_version: "encryptionkey@0"
I20260925 01:16:57.294404 23526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:16:57.297657 23541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:57.298882 23526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:16:57.299057 23526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "f9a0fa48d31346f99dbfe32700c077e0"
format_stamp: "Formatted at 2026-09-25 01:16:57 on dist-test-slave-s12s"
server_key: "a5f1fc80443e8a560fcd2a87f4d022d2"
server_key_iv: "bbab97dfe33b2fa9b2693764d4cef7db"
server_key_version: "encryptionkey@0"
I20260925 01:16:57.299250 23526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:57.313602 23526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:57.318320 23526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:57.318612 23526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:57.319545 23526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:16:57.320850 23526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:57.320921 23526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:57.321110 23526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:57.321168 23526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:57.351904 23526 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:36273
I20260925 01:16:57.351943 23653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:36273 every 8 connection(s)
I20260925 01:16:57.353210 23526 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:16:57.359699 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 23526
I20260925 01:16:57.359856 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:16:57.360352 20890 external_mini_cluster.cc:1428] Setting key 8fdbd6aa6e14a07c25e700addefa08f8
Sep 25 01:16:57 dist-test-slave-s12s krb5kdc[23286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:57.374370 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:57.355665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:42217 (local address 127.20.102.190:45537)
0925 01:16:57.355887 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:57.355893 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:57.357027 (+ 1134us) server_negotiation.cc:407] Connection header received
0925 01:16:57.358752 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:57.358757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:57.358823 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:57.358909 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:57.360495 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:57.361402 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:57.363272 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:57.363700 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:57.363794 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:57.367866 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:57.367871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:57.367874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:57.367905 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:57.371285 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:57.371924 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:57.371932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:57.371939 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:57.372097 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:57.372471 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:57.372478 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:57.372485 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:57.372912 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:57.373063 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:57.373686 (+ 623us) server_negotiation.cc:299] Negotiation successful
0925 01:16:57.373935 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 01:16:57.374917 23656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:57.356151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:45537 (local address 127.20.102.130:42217)
0925 01:16:57.356880 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:57.356921 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:57.358513 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:57.359105 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:57.359112 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:57.359630 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:57.360330 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:57.360347 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:57.361554 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:57.361562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:57.363120 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:57.363137 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:57.363928 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:57.363938 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:57.364300 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:57.365095 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:57.365129 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:57.367652 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:57.371418 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:57.371424 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:57.371434 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:57.371806 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:57.372202 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:57.372205 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:57.372207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:57.372367 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:57.373050 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:57.373056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:57.373325 (+ 269us) client_negotiation.cc:769] Sending connection context
0925 01:16:57.373554 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 01:16:57.373890 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":252,"threads_started":1}
I20260925 01:16:57.376356 23654 heartbeater.cc:347] Connected to a master server at 127.20.102.190:45537
I20260925 01:16:57.376694 23654 heartbeater.cc:464] Registering TS with master...
I20260925 01:16:57.377628 23654 heartbeater.cc:511] Master 127.20.102.190:45537 requested a full tablet report, sending...
I20260925 01:16:57.379395 23331 ts_manager.cc:194] Registered new tserver with Master: f9a0fa48d31346f99dbfe32700c077e0 (127.20.102.130:36273)
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
I20260925 01:16:57.380214 23331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:42217
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:16:57.404428 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:45537
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:57.726271 23659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:57.726591 23659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:57.726758 23659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:57.734011 23659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:57.734158 23659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:57.734328 23659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:16:57.743813 23659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:45537
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:57.745105 23659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:57.746402 23659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:57.759341 23664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:57.759430 23665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:57.759946 23667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:57.760495 23659 server_base.cc:1034] running on GCE node
I20260925 01:16:57.761139 23659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:57.762001 23659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:57.763247 23659 hybrid_clock.cc:648] HybridClock initialized: now 1790299017763189 us; error 49 us; skew 500 ppm
I20260925 01:16:57.763578 23659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:57 dist-test-slave-s12s krb5kdc[23286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:57.766659 23659 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:16:57.768024 23659 webserver.cc:460] Webserver started at http://127.20.102.131:34681/ using document root <none> and password file <none>
I20260925 01:16:57.768739 23659 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:57.768879 23659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:57.769186 23659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:57.771996 23659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "82bcb926e9c2403eb2bc9a594266cf67"
format_stamp: "Formatted at 2026-09-25 01:16:57 on dist-test-slave-s12s"
server_key: "b99e9f164d8d88fbe13c5dd36acda60b"
server_key_iv: "0f308d06261b05aff09c654177dee072"
server_key_version: "encryptionkey@0"
I20260925 01:16:57.772835 23659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "82bcb926e9c2403eb2bc9a594266cf67"
format_stamp: "Formatted at 2026-09-25 01:16:57 on dist-test-slave-s12s"
server_key: "b99e9f164d8d88fbe13c5dd36acda60b"
server_key_iv: "0f308d06261b05aff09c654177dee072"
server_key_version: "encryptionkey@0"
I20260925 01:16:57.777667 23659 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 01:16:57.780874 23674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:57.782181 23659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:16:57.782343 23659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "82bcb926e9c2403eb2bc9a594266cf67"
format_stamp: "Formatted at 2026-09-25 01:16:57 on dist-test-slave-s12s"
server_key: "b99e9f164d8d88fbe13c5dd36acda60b"
server_key_iv: "0f308d06261b05aff09c654177dee072"
server_key_version: "encryptionkey@0"
I20260925 01:16:57.782533 23659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:57.808511 23659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:57.812860 23659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:57.813141 23659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:57.814018 23659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:16:57.815447 23659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:16:57.815546 23659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:57.815688 23659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:16:57.815760 23659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:57.848627 23659 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:37121
I20260925 01:16:57.848731 23786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:37121 every 8 connection(s)
I20260925 01:16:57.849889 23659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:16:57.851552 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 23659
I20260925 01:16:57.851716 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:16:57.852167 20890 external_mini_cluster.cc:1428] Setting key 93b4b53c67a7a2d1cb1677f940e78c21
Sep 25 01:16:57 dist-test-slave-s12s krb5kdc[23286](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299017, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:57.870359 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:57.852564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:52387 (local address 127.20.102.190:45537)
0925 01:16:57.852837 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:57.852843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:57.854196 (+ 1353us) server_negotiation.cc:407] Connection header received
0925 01:16:57.856184 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:57.856188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:57.856252 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:57.856335 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:57.857820 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:57.858603 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:57.860095 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:57.860831 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:57.860973 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:57.864420 (+ 3447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:57.864425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:57.864428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:57.864459 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:57.867557 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:57.868258 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:57.868262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:57.868264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:57.868391 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:57.868946 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:57.868949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:57.868951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:57.869208 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:57.869313 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:57.869970 (+ 657us) server_negotiation.cc:299] Negotiation successful
0925 01:16:57.870130 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260925 01:16:57.871405 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:57.853147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:45537 (local address 127.20.102.131:52387)
0925 01:16:57.854005 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0925 01:16:57.854075 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 01:16:57.855946 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:16:57.856567 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:16:57.856574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:16:57.857124 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:16:57.857668 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:57.857685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:57.858769 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:57.858773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:57.859944 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:16:57.859963 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:16:57.861048 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:16:57.861053 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:16:57.861242 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:57.861904 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:16:57.861926 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:16:57.864247 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:16:57.867694 (+ 3447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:57.867701 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:57.867710 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:57.868160 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:57.868487 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:16:57.868491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:16:57.868493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:16:57.868828 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:16:57.869318 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:16:57.869323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:16:57.869623 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 01:16:57.869874 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 01:16:57.870237 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":432,"threads_started":1}
I20260925 01:16:57.872846 23787 heartbeater.cc:347] Connected to a master server at 127.20.102.190:45537
I20260925 01:16:57.873157 23787 heartbeater.cc:464] Registering TS with master...
I20260925 01:16:57.873828 23787 heartbeater.cc:511] Master 127.20.102.190:45537 requested a full tablet report, sending...
I20260925 01:16:57.875211 23331 ts_manager.cc:194] Registered new tserver with Master: 82bcb926e9c2403eb2bc9a594266cf67 (127.20.102.131:37121)
I20260925 01:16:57.876025 23331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:52387
I20260925 01:16:57.879678 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:16:57.891949 23520 heartbeater.cc:503] Master 127.20.102.190:45537 was elected leader, sending a full tablet report...
W20260925 01:16:57.909142 23522 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:16:57.906849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.20.102.190:45537)
0925 01:16:57.907024 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:57.907028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:57.907444 (+ 416us) server_negotiation.cc:407] Connection header received
0925 01:16:57.907597 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:57.907601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:57.907661 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:57.907744 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:57.908909 (+ 1165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41130: BlockingRecv error: recv got EOF from 127.0.0.1:41130 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:16:57.912469 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 23392
W20260925 01:16:57.915220 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 23392: Not authorized: Client connection negotiation failed: client connection to 127.20.102.129:44065: server requires authentication, but client does not have Kerberos credentials available
I20260925 01:16:57.915428 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 23392
I20260925 01:16:57.931746 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 23526
W20260925 01:16:57.934293 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 23526: Not authorized: Client connection negotiation failed: client connection to 127.20.102.130:36273: server requires authentication, but client does not have Kerberos credentials available
I20260925 01:16:57.934430 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 23526
I20260925 01:16:57.949970 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 23659
W20260925 01:16:57.952826 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 23659: Not authorized: Client connection negotiation failed: client connection to 127.20.102.131:37121: server requires authentication, but client does not have Kerberos credentials available
I20260925 01:16:57.952984 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 23659
I20260925 01:16:57.965453 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 23299
I20260925 01:16:57.966600 23361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:16:58.068858 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 23299
2026-09-25T01:16:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3493 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:16:58 dist-test-slave-s12s krb5kdc[23804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:16:58 dist-test-slave-s12s krb5kdc[23804](info): set up 2 sockets
Sep 25 01:16:58 dist-test-slave-s12s krb5kdc[23804](info): commencing operation
krb5kdc: starting...
W20260925 01:16:59.149163 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:16:59 dist-test-slave-s12s krb5kdc[23804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:16:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:16:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:16:59.339955 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:36853
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:37835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:36853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:16:59.657902 23817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:16:59.658219 23817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:16:59.658385 23817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:16:59.665372 23817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:16:59.665506 23817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:16:59.665621 23817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:16:59.665714 23817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:16:59.665808 23817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:16:59.676450 23817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:37835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:36853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:36853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:16:59.677791 23817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:16:59.679051 23817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:16:59.690768 23822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:59.690837 23825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:16:59.692010 23823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:16:59.692915 23817 server_base.cc:1034] running on GCE node
I20260925 01:16:59.693740 23817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:16:59.695094 23817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:16:59.696293 23817 hybrid_clock.cc:648] HybridClock initialized: now 1790299019696244 us; error 42 us; skew 500 ppm
I20260925 01:16:59.696630 23817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:16:59 dist-test-slave-s12s krb5kdc[23804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299019, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:16:59.704095 23817 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:16:59.705619 23817 webserver.cc:460] Webserver started at http://127.20.102.190:43571/ using document root <none> and password file <none>
I20260925 01:16:59.706327 23817 fs_manager.cc:359] Metadata directory not provided
I20260925 01:16:59.706471 23817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:16:59.706792 23817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:16:59.709797 23817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "4d5a3aa800bb4aeba9ecd06ff8971713"
format_stamp: "Formatted at 2026-09-25 01:16:59 on dist-test-slave-s12s"
server_key: "5c13bad73b9036f7fb4b0b46eac25a85"
server_key_iv: "29d23476bf0896ce736b1702964f4a3e"
server_key_version: "encryptionkey@0"
I20260925 01:16:59.710580 23817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "4d5a3aa800bb4aeba9ecd06ff8971713"
format_stamp: "Formatted at 2026-09-25 01:16:59 on dist-test-slave-s12s"
server_key: "5c13bad73b9036f7fb4b0b46eac25a85"
server_key_iv: "29d23476bf0896ce736b1702964f4a3e"
server_key_version: "encryptionkey@0"
I20260925 01:16:59.715485 23817 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:16:59.718748 23832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:16:59.720010 23817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:16:59.720216 23817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "4d5a3aa800bb4aeba9ecd06ff8971713"
format_stamp: "Formatted at 2026-09-25 01:16:59 on dist-test-slave-s12s"
server_key: "5c13bad73b9036f7fb4b0b46eac25a85"
server_key_iv: "29d23476bf0896ce736b1702964f4a3e"
server_key_version: "encryptionkey@0"
I20260925 01:16:59.720405 23817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:16:59.734055 23817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:16:59.738296 23817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:16:59.738559 23817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:16:59.757516 23817 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:36853
I20260925 01:16:59.757555 23883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:36853 every 8 connection(s)
I20260925 01:16:59.758865 23817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:16:59.762660 23884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:16:59.767712 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 23817
I20260925 01:16:59.767997 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:16:59.768781 20890 external_mini_cluster.cc:1428] Setting key 763990fd11ba1cddd161216cc0e870af
I20260925 01:16:59.774683 23884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713: Bootstrap starting.
I20260925 01:16:59.778477 23884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713: Neither blocks nor log segments found. Creating new log.
I20260925 01:16:59.779692 23884 log.cc:824] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713: Log is configured to *not* fsync() on all Append() calls
I20260925 01:16:59.783576 23884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713: No bootstrap required, opened a new log
Sep 25 01:16:59 dist-test-slave-s12s krb5kdc[23804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:16:59.792938 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:59.770707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.20.102.190:36853)
0925 01:16:59.771613 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:59.771631 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:59.771676 (+ 45us) server_negotiation.cc:407] Connection header received
0925 01:16:59.773536 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:59.773559 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:59.773958 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:59.774292 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:59.776140 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:59.777534 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:59.779283 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:59.780024 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:59.780197 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:59.785439 (+ 5242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:59.785447 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:59.785464 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:16:59.785498 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:59.788850 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:59.789578 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:59.789592 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:59.789599 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:59.789767 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:16:59.790160 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:16:59.790165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:16:59.790175 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:16:59.790705 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:59.791023 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:59.791470 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 01:16:59.791884 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":451,"threads_started":1}
I20260925 01:16:59.793156 23884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5a3aa800bb4aeba9ecd06ff8971713" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36853 } }
I20260925 01:16:59.793622 23884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:16:59.793766 23884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d5a3aa800bb4aeba9ecd06ff8971713, State: Initialized, Role: FOLLOWER
I20260925 01:16:59.794450 23884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5a3aa800bb4aeba9ecd06ff8971713" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36853 } }
I20260925 01:16:59.794756 23884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:16:59.794934 23884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:16:59.795192 23884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:16:59.797405 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:16:59.798837 23884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5a3aa800bb4aeba9ecd06ff8971713" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36853 } }
I20260925 01:16:59.799532 23884 leader_election.cc:302] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d5a3aa800bb4aeba9ecd06ff8971713; no voters:
I20260925 01:16:59.800271 23884 leader_election.cc:288] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:16:59.800319 23889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:16:59.802883 23884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:16:59.806916 23889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [term 1 LEADER]: Becoming Leader. State: Replica: 4d5a3aa800bb4aeba9ecd06ff8971713, State: Running, Role: LEADER
I20260925 01:16:59.807586 23889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5a3aa800bb4aeba9ecd06ff8971713" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36853 } }
I20260925 01:16:59.812809 23890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d5a3aa800bb4aeba9ecd06ff8971713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5a3aa800bb4aeba9ecd06ff8971713" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36853 } } }
I20260925 01:16:59.813179 23890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [sys.catalog]: This master's current role is: LEADER
I20260925 01:16:59.818182 23891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d5a3aa800bb4aeba9ecd06ff8971713. Latest consensus state: current_term: 1 leader_uuid: "4d5a3aa800bb4aeba9ecd06ff8971713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5a3aa800bb4aeba9ecd06ff8971713" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36853 } } }
I20260925 01:16:59.818604 23891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713 [sys.catalog]: This master's current role is: LEADER
I20260925 01:16:59.818665 23903 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:16:59.822897 23903 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:16:59.832659 23903 catalog_manager.cc:1347] Generated new cluster ID: 04c0366508ea447dbd064af1c017ab97
I20260925 01:16:59.832851 23903 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:16:59.849140 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:16:59.840621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60580 (local address 127.20.102.190:36853)
0925 01:16:59.840852 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 01:16:59.840861 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:16:59.841420 (+ 559us) server_negotiation.cc:407] Connection header received
0925 01:16:59.841632 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:16:59.841636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:16:59.841697 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:16:59.841801 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:16:59.842794 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:59.843690 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:59.845055 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:16:59.845476 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:16:59.845608 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:16:59.846675 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:16:59.846680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:16:59.846683 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:16:59.847273 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:16:59.847329 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:16:59.847356 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:16:59.847472 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 01:16:59.848742 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0925 01:16:59.848845 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":74}
I20260925 01:16:59.857158 23903 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:16:59.859119 23903 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:16:59.874148 23903 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d5a3aa800bb4aeba9ecd06ff8971713: Generated new TSK 0
I20260925 01:16:59.874936 23903 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:16:59.881737 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 23817
I20260925 01:16:59.883163 23879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:16:59.970714 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 23817
2026-09-25T01:16:59Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1896 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:17:00 dist-test-slave-s12s krb5kdc[23919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:17:00 dist-test-slave-s12s krb5kdc[23919](info): set up 2 sockets
Sep 25 01:17:00 dist-test-slave-s12s krb5kdc[23919](info): commencing operation
krb5kdc: starting...
W20260925 01:17:01.046998 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:17:01 dist-test-slave-s12s krb5kdc[23919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:17:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:17:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:17:01.239761 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:35061
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:35061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:01.553149 23932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:01.553472 23932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:01.553639 23932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:01.560736 23932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:01.560889 23932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:01.561004 23932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:01.561107 23932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:01.561193 23932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:17:01.571252 23932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:35061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:01.572712 23932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:01.574035 23932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:01.585543 23940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:01.585592 23938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:01.585629 23937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:01.586454 23932 server_base.cc:1034] running on GCE node
I20260925 01:17:01.587289 23932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:01.588603 23932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:01.589808 23932 hybrid_clock.cc:648] HybridClock initialized: now 1790299021589763 us; error 29 us; skew 500 ppm
I20260925 01:17:01.590167 23932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:01 dist-test-slave-s12s krb5kdc[23919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299021, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:01.593379 23932 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:17:01.594858 23932 webserver.cc:460] Webserver started at http://127.20.102.190:41547/ using document root <none> and password file <none>
I20260925 01:17:01.595520 23932 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:01.595661 23932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:01.595980 23932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:01.598937 23932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "609a60dbfd814c9cad55042e79ba6232"
format_stamp: "Formatted at 2026-09-25 01:17:01 on dist-test-slave-s12s"
server_key: "54226a3e00d228a93b6b410f0389df5b"
server_key_iv: "bc641832cd24129d24bbed46a489d53c"
server_key_version: "encryptionkey@0"
I20260925 01:17:01.599738 23932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "609a60dbfd814c9cad55042e79ba6232"
format_stamp: "Formatted at 2026-09-25 01:17:01 on dist-test-slave-s12s"
server_key: "54226a3e00d228a93b6b410f0389df5b"
server_key_iv: "bc641832cd24129d24bbed46a489d53c"
server_key_version: "encryptionkey@0"
I20260925 01:17:01.604737 23932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260925 01:17:01.608060 23947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:01.609591 23932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:01.609782 23932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "609a60dbfd814c9cad55042e79ba6232"
format_stamp: "Formatted at 2026-09-25 01:17:01 on dist-test-slave-s12s"
server_key: "54226a3e00d228a93b6b410f0389df5b"
server_key_iv: "bc641832cd24129d24bbed46a489d53c"
server_key_version: "encryptionkey@0"
I20260925 01:17:01.609982 23932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:01.647946 23932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:01.652766 23932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:01.653055 23932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:01.671659 23932 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:35061
I20260925 01:17:01.671692 23998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:35061 every 8 connection(s)
I20260925 01:17:01.673094 23932 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:17:01.676909 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 23932
I20260925 01:17:01.676995 23999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:01.677129 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:17:01.677919 20890 external_mini_cluster.cc:1428] Setting key 7e0840142af8028311416b2529a3f571
I20260925 01:17:01.688990 23999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232: Bootstrap starting.
Sep 25 01:17:01 dist-test-slave-s12s krb5kdc[23919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:01.692150 23999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:01.693565 23999 log.cc:824] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:01.696996 23999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232: No bootstrap required, opened a new log
I20260925 01:17:01.701761 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:01.680373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48892 (local address 127.20.102.190:35061)
0925 01:17:01.681146 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:01.681167 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:01.681209 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:17:01.683083 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:01.683104 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:01.683490 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:01.683879 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:01.685344 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:01.686692 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:01.687965 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:01.688630 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:01.688889 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:01.693408 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:01.693417 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:01.693440 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:01.693480 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:01.697178 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:01.698097 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:01.698104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:01.698111 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:01.698338 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:01.698802 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:01.698805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:01.698807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:01.699407 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:01.699705 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:01.700178 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 01:17:01.700637 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":430,"threads_started":1}
I20260925 01:17:01.705595 23999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609a60dbfd814c9cad55042e79ba6232" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35061 } }
I20260925 01:17:01.705945 23999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:01.706041 23999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609a60dbfd814c9cad55042e79ba6232, State: Initialized, Role: FOLLOWER
I20260925 01:17:01.706596 23999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609a60dbfd814c9cad55042e79ba6232" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35061 } }
I20260925 01:17:01.706826 23999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:17:01.706971 23999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:17:01.707126 23999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:01.709372 23999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609a60dbfd814c9cad55042e79ba6232" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35061 } }
I20260925 01:17:01.709844 23999 leader_election.cc:302] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 609a60dbfd814c9cad55042e79ba6232; no voters:
I20260925 01:17:01.710405 23999 leader_election.cc:288] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:17:01.710609 24004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:01.711346 24004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [term 1 LEADER]: Becoming Leader. State: Replica: 609a60dbfd814c9cad55042e79ba6232, State: Running, Role: LEADER
I20260925 01:17:01.711938 24004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609a60dbfd814c9cad55042e79ba6232" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35061 } }
I20260925 01:17:01.712610 23999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:01.715047 24006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 609a60dbfd814c9cad55042e79ba6232. Latest consensus state: current_term: 1 leader_uuid: "609a60dbfd814c9cad55042e79ba6232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609a60dbfd814c9cad55042e79ba6232" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35061 } } }
I20260925 01:17:01.715480 24005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "609a60dbfd814c9cad55042e79ba6232" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609a60dbfd814c9cad55042e79ba6232" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35061 } } }
I20260925 01:17:01.715719 24006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:01.715806 24005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:01.716776 24011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:17:01.723026 24011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:17:01.731324 24011 catalog_manager.cc:1347] Generated new cluster ID: 05432bdfeea546d79196c4e4dfa17af9
I20260925 01:17:01.731427 24011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:17:01.759281 24011 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:17:01.761250 24011 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:17:01.781822 24011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 609a60dbfd814c9cad55042e79ba6232: Generated new TSK 0
I20260925 01:17:01.782858 24011 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:17:01.835899 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35061
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:02.160812 24025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:02.161158 24025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:02.161332 24025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:02.168529 24025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:02.168673 24025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:02.168913 24025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:17:02.178604 24025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:02.179909 24025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:02.181435 24025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:02.194082 24031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:02.194252 24030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:02.194092 24033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:02.195075 24025 server_base.cc:1034] running on GCE node
I20260925 01:17:02.195772 24025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:02.196594 24025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:02.197798 24025 hybrid_clock.cc:648] HybridClock initialized: now 1790299022197754 us; error 29 us; skew 500 ppm
I20260925 01:17:02.198123 24025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:02 dist-test-slave-s12s krb5kdc[23919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299022, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:02.201268 24025 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:17:02.202657 24025 webserver.cc:460] Webserver started at http://127.20.102.129:33063/ using document root <none> and password file <none>
I20260925 01:17:02.203342 24025 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:02.203475 24025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:02.203790 24025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:02.206713 24025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "47e0099144a446a690297b27a792f5d6"
format_stamp: "Formatted at 2026-09-25 01:17:02 on dist-test-slave-s12s"
server_key: "68d1a6a54d33d6de6549c65d67475fa4"
server_key_iv: "0666c08213f035d80fb19ede2e50999f"
server_key_version: "encryptionkey@0"
I20260925 01:17:02.207515 24025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "47e0099144a446a690297b27a792f5d6"
format_stamp: "Formatted at 2026-09-25 01:17:02 on dist-test-slave-s12s"
server_key: "68d1a6a54d33d6de6549c65d67475fa4"
server_key_iv: "0666c08213f035d80fb19ede2e50999f"
server_key_version: "encryptionkey@0"
I20260925 01:17:02.212777 24025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 01:17:02.216575 24040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:02.217939 24025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:02.218115 24025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "47e0099144a446a690297b27a792f5d6"
format_stamp: "Formatted at 2026-09-25 01:17:02 on dist-test-slave-s12s"
server_key: "68d1a6a54d33d6de6549c65d67475fa4"
server_key_iv: "0666c08213f035d80fb19ede2e50999f"
server_key_version: "encryptionkey@0"
I20260925 01:17:02.218330 24025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:02.237618 24025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:02.241963 24025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:02.242271 24025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:02.243244 24025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:02.244571 24025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:02.244644 24025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:02.244822 24025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:02.244863 24025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:02.278362 24025 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:42317
I20260925 01:17:02.278398 24152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:42317 every 8 connection(s)
I20260925 01:17:02.279656 24025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:17:02.286290 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 24025
I20260925 01:17:02.286511 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:17:02.287163 20890 external_mini_cluster.cc:1428] Setting key 42fb8c8f6719fcf44f63ec774d6d758e
Sep 25 01:17:02 dist-test-slave-s12s krb5kdc[23919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299022, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:02.302013 24155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:02.282354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:37637 (local address 127.20.102.190:35061)
0925 01:17:02.282821 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:02.282825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:02.283676 (+ 851us) server_negotiation.cc:407] Connection header received
0925 01:17:02.285600 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:02.285606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:02.285697 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:02.285818 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:02.287429 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:02.288409 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:02.290909 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:02.291323 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:02.291420 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:02.295963 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:02.295968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:02.295971 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:02.296001 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:02.298566 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:02.299589 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:02.299593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:02.299595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:02.299707 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:02.300249 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:02.300252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:02.300254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:02.300494 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:02.300622 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:02.301595 (+ 973us) server_negotiation.cc:299] Negotiation successful
0925 01:17:02.301756 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":283,"threads_started":1}
I20260925 01:17:02.303793 24156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:02.282774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35061 (local address 127.20.102.129:37637)
0925 01:17:02.283519 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:02.283557 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:02.285314 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:02.286054 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:02.286063 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:02.286646 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:02.287232 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:02.287274 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:02.288628 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:02.288632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:02.290737 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:02.290756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:02.291566 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:02.291576 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:02.291897 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:02.292724 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:02.292769 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:02.295722 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:02.298743 (+ 3021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:02.298756 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:02.298777 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:02.299446 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:02.299839 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:02.299846 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:02.299853 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:02.300129 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:02.300628 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:02.300639 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:02.301143 (+ 504us) client_negotiation.cc:769] Sending connection context
0925 01:17:02.301504 (+ 361us) client_negotiation.cc:240] Negotiation successful
0925 01:17:02.302127 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":261,"threads_started":1}
I20260925 01:17:02.305498 24153 heartbeater.cc:347] Connected to a master server at 127.20.102.190:35061
I20260925 01:17:02.305878 24153 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:02.306945 24153 heartbeater.cc:511] Master 127.20.102.190:35061 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
I20260925 01:17:02.309278 23964 ts_manager.cc:194] Registered new tserver with Master: 47e0099144a446a690297b27a792f5d6 (127.20.102.129:42317)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
I20260925 01:17:02.310935 23964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:37637
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:17:02.334311 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35061
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:02.654698 24159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:02.655030 24159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:02.655220 24159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:02.662444 24159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:02.662592 24159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:02.662762 24159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:17:02.672300 24159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:02.673588 24159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:02.674888 24159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:02.689205 24164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:02.689242 24165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:02.689761 24167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:02.690090 24159 server_base.cc:1034] running on GCE node
I20260925 01:17:02.690703 24159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:02.691618 24159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:02.692799 24159 hybrid_clock.cc:648] HybridClock initialized: now 1790299022692735 us; error 37 us; skew 500 ppm
I20260925 01:17:02.693204 24159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:02 dist-test-slave-s12s krb5kdc[23919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299022, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:02.696326 24159 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:17:02.697749 24159 webserver.cc:460] Webserver started at http://127.20.102.130:34875/ using document root <none> and password file <none>
I20260925 01:17:02.698400 24159 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:02.698503 24159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:02.698753 24159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:02.701560 24159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "aba1370bc849407bac08af9c270e4978"
format_stamp: "Formatted at 2026-09-25 01:17:02 on dist-test-slave-s12s"
server_key: "857f1ea58f37c8dc80e55376714edafe"
server_key_iv: "83d269f69450115a697ce01e411e989e"
server_key_version: "encryptionkey@0"
I20260925 01:17:02.702256 24159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "aba1370bc849407bac08af9c270e4978"
format_stamp: "Formatted at 2026-09-25 01:17:02 on dist-test-slave-s12s"
server_key: "857f1ea58f37c8dc80e55376714edafe"
server_key_iv: "83d269f69450115a697ce01e411e989e"
server_key_version: "encryptionkey@0"
I20260925 01:17:02.707094 24159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:17:02.710394 24174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:02.711665 24159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:17:02.711791 24159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "aba1370bc849407bac08af9c270e4978"
format_stamp: "Formatted at 2026-09-25 01:17:02 on dist-test-slave-s12s"
server_key: "857f1ea58f37c8dc80e55376714edafe"
server_key_iv: "83d269f69450115a697ce01e411e989e"
server_key_version: "encryptionkey@0"
I20260925 01:17:02.711951 24159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:02.734251 24159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:02.737989 24159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:02.738238 24159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:02.739151 24159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:02.740396 24159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:02.740466 24159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:02.740546 24159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:02.740589 24159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:02.773487 24159 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:35699
I20260925 01:17:02.773540 24286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:35699 every 8 connection(s)
I20260925 01:17:02.774814 24159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:17:02.782214 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 24159
I20260925 01:17:02.782421 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:17:02.782851 20890 external_mini_cluster.cc:1428] Setting key af55348fa51de2f6aacf795c5b64f0d4
Sep 25 01:17:02 dist-test-slave-s12s krb5kdc[23919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299022, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:02.796159 24155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:02.777407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:49975 (local address 127.20.102.190:35061)
0925 01:17:02.777590 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:02.777595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:02.778714 (+ 1119us) server_negotiation.cc:407] Connection header received
0925 01:17:02.780562 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:02.780568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:02.780661 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:02.780817 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:02.782678 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:02.783457 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:02.785350 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:02.785736 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:02.785931 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:02.789739 (+ 3808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:02.789748 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:02.789756 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:02.789804 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:02.792880 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:02.793803 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:02.793811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:02.793815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:02.793951 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:02.794370 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:02.794376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:02.794379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:02.794658 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:02.794817 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:02.795539 (+ 722us) server_negotiation.cc:299] Negotiation successful
0925 01:17:02.795779 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:17:02.797225 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:02.777866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35061 (local address 127.20.102.130:49975)
0925 01:17:02.778551 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:02.778598 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:02.780285 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:02.781058 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:02.781066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:02.781636 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:02.782499 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:02.782527 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:02.783607 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:02.783612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:02.785205 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:02.785216 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:02.786016 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:02.786023 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:02.786264 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:02.786954 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:02.786976 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:02.789496 (+ 2520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:02.793019 (+ 3523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:02.793028 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:02.793045 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:02.793688 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:02.794063 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:02.794069 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:02.794072 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:02.794257 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:02.794789 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:02.794796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:02.795188 (+ 392us) client_negotiation.cc:769] Sending connection context
0925 01:17:02.795460 (+ 272us) client_negotiation.cc:240] Negotiation successful
0925 01:17:02.796009 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":257,"threads_started":1}
I20260925 01:17:02.798749 24287 heartbeater.cc:347] Connected to a master server at 127.20.102.190:35061
I20260925 01:17:02.799130 24287 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:02.800114 24287 heartbeater.cc:511] Master 127.20.102.190:35061 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
I20260925 01:17:02.801926 23964 ts_manager.cc:194] Registered new tserver with Master: aba1370bc849407bac08af9c270e4978 (127.20.102.130:35699)
I20260925 01:17:02.802800 23964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:49975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
W20260925 01:17:02.911208 20890 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.20.102.131: real 0.126s user 0.001s sys 0.009s
I20260925 01:17:02.911866 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35061
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:03.234663 24292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:03.234961 24292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:03.235148 24292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:03.243849 24292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:03.243989 24292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:03.244171 24292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:17:03.253609 24292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:03.254845 24292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:03.256167 24292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:03.268846 24300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:03.268905 24297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:03.268929 24298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:03.269824 24292 server_base.cc:1034] running on GCE node
I20260925 01:17:03.270344 24292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:03.271286 24292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:03.272490 24292 hybrid_clock.cc:648] HybridClock initialized: now 1790299023272415 us; error 69 us; skew 500 ppm
I20260925 01:17:03.272837 24292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:03 dist-test-slave-s12s krb5kdc[23919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:03.275879 24292 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:17:03.277341 24292 webserver.cc:460] Webserver started at http://127.20.102.131:42261/ using document root <none> and password file <none>
I20260925 01:17:03.278002 24292 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:03.278147 24292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:03.278520 24292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:03.281391 24292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "c88d84a3d2c348f1a280f9cc040f2ddc"
format_stamp: "Formatted at 2026-09-25 01:17:03 on dist-test-slave-s12s"
server_key: "1ad887e66466b64fb9f75e69e34d5ee7"
server_key_iv: "c89498a62b8f7c2826a82fc48a48e26f"
server_key_version: "encryptionkey@0"
I20260925 01:17:03.282171 24292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "c88d84a3d2c348f1a280f9cc040f2ddc"
format_stamp: "Formatted at 2026-09-25 01:17:03 on dist-test-slave-s12s"
server_key: "1ad887e66466b64fb9f75e69e34d5ee7"
server_key_iv: "c89498a62b8f7c2826a82fc48a48e26f"
server_key_version: "encryptionkey@0"
I20260925 01:17:03.287168 24292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:17:03.290478 24307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:03.291697 24292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:17:03.291865 24292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "c88d84a3d2c348f1a280f9cc040f2ddc"
format_stamp: "Formatted at 2026-09-25 01:17:03 on dist-test-slave-s12s"
server_key: "1ad887e66466b64fb9f75e69e34d5ee7"
server_key_iv: "c89498a62b8f7c2826a82fc48a48e26f"
server_key_version: "encryptionkey@0"
I20260925 01:17:03.292044 24292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:03.314284 24153 heartbeater.cc:503] Master 127.20.102.190:35061 was elected leader, sending a full tablet report...
I20260925 01:17:03.322332 24292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:03.326259 24292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:03.326568 24292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:03.327491 24292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:03.328848 24292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:03.328919 24292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:03.329051 24292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:03.329115 24292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:03.362201 24292 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:37381
I20260925 01:17:03.362242 24419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:37381 every 8 connection(s)
I20260925 01:17:03.363481 24292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:17:03.370127 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 24292
I20260925 01:17:03.370302 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:17:03.370764 20890 external_mini_cluster.cc:1428] Setting key 30f2adcc4e4c9c6593dd7443c96774cd
Sep 25 01:17:03 dist-test-slave-s12s krb5kdc[23919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299023, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:03.382889 24422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:03.366065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:40481 (local address 127.20.102.190:35061)
0925 01:17:03.366487 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:03.366492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:03.367495 (+ 1003us) server_negotiation.cc:407] Connection header received
0925 01:17:03.369800 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:03.369806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:03.369898 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:03.370029 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:03.371626 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:03.372443 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:03.374002 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:03.374404 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:03.374549 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:03.377941 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:03.377946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:03.377949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:03.377980 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:03.380306 (+ 2326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:03.381009 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:03.381012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:03.381015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:03.381131 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:03.381496 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:03.381499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:03.381501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:03.381746 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:03.381850 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:03.382466 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 01:17:03.382631 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":199,"threads_started":1}
I20260925 01:17:03.383881 24423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:03.366659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35061 (local address 127.20.102.131:40481)
0925 01:17:03.367344 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:03.367380 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:03.369533 (+ 2153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:03.370296 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:03.370306 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:03.370875 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:03.371454 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:03.371470 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:03.372591 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:03.372596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:03.373857 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:03.373868 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:03.374620 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:03.374628 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:03.374870 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:03.375467 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:03.375488 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:03.377767 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:03.380435 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:03.380441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:03.380452 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:03.380902 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:03.381222 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:03.381224 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:03.381227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:03.381404 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:03.381864 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:03.381869 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:03.382170 (+ 301us) client_negotiation.cc:769] Sending connection context
0925 01:17:03.382374 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 01:17:03.382715 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":270,"threads_started":1}
I20260925 01:17:03.385306 24420 heartbeater.cc:347] Connected to a master server at 127.20.102.190:35061
I20260925 01:17:03.385680 24420 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:03.386447 24420 heartbeater.cc:511] Master 127.20.102.190:35061 requested a full tablet report, sending...
I20260925 01:17:03.387836 23964 ts_manager.cc:194] Registered new tserver with Master: c88d84a3d2c348f1a280f9cc040f2ddc (127.20.102.131:37381)
I20260925 01:17:03.388607 23964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:40481
I20260925 01:17:03.398236 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:17:03 dist-test-slave-s12s krb5kdc[23919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299023, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 01:17:03 dist-test-slave-s12s krb5kdc[23919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299023, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:03.447247 24422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:03.433032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.20.102.190:35061)
0925 01:17:03.433220 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:03.433224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:03.433700 (+ 476us) server_negotiation.cc:407] Connection header received
0925 01:17:03.433856 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:03.433860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:03.433928 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:03.434054 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:03.435357 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:03.436097 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:03.437315 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:03.437738 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:03.437838 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:03.441684 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:03.441689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:03.441692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:03.441728 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:03.444568 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:03.445294 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:03.445298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:03.445300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:03.445436 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:03.445868 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:03.445873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:03.445875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:03.446282 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:03.446524 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:03.446796 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 01:17:03.446998 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 01:17:03.447932 23964 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49130
I20260925 01:17:03.449767 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 24025
Sep 25 01:17:03 dist-test-slave-s12s krb5kdc[23919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299023, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:17:03.467859 24432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:03.450739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35576 (local address 127.20.102.129:42317)
0925 01:17:03.453016 (+ 2277us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:03.453023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:03.453052 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:17:03.453123 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:03.453129 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:03.453384 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:03.453530 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:03.454653 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:03.455729 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:03.457120 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:03.458181 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:03.458298 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:03.462118 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:03.462131 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:03.462147 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:03.462226 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:03.465262 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:03.465886 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:03.465894 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:03.465896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:03.466028 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:03.466380 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:03.466384 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:03.466386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:03.466769 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:03.467024 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:03.467204 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 01:17:03.467411 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2099,"thread_start_us":1889,"threads_started":1}
W20260925 01:17:03.469349 24148 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35576
W20260925 01:17:03.470566 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 24025: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:17:03.470778 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 24025
I20260925 01:17:03.487820 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 24159
Sep 25 01:17:03 dist-test-slave-s12s krb5kdc[23919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299023, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:17:03.502967 24433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:03.488631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34752 (local address 127.20.102.130:35699)
0925 01:17:03.489275 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:03.489281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:03.489307 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:17:03.489365 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:03.489372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:03.489598 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:03.489722 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:03.490821 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:03.491773 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:03.492945 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:03.493519 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:03.493615 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:03.497766 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:03.497776 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:03.497789 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:03.497828 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:03.500473 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:03.501121 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:03.501128 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:03.501130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:03.501256 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:03.501571 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:03.501574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:03.501576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:03.501898 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:03.502123 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:03.502368 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 01:17:03.502607 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":302,"threads_started":1}
W20260925 01:17:03.504251 24282 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34752
W20260925 01:17:03.505461 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 24159: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:17:03.505623 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 24159
I20260925 01:17:03.522213 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 24292
Sep 25 01:17:03 dist-test-slave-s12s krb5kdc[23919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299023, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:17:03.539727 24434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:03.523184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47036 (local address 127.20.102.131:37381)
0925 01:17:03.523937 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:03.523946 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:03.523981 (+ 35us) server_negotiation.cc:407] Connection header received
0925 01:17:03.524047 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:03.524053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:03.524265 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:03.524493 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:03.525938 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:03.527025 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:03.528373 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:03.529089 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:03.529237 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:03.533734 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:03.533746 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:03.533760 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:03.533794 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:03.536564 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:03.537388 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:03.537403 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:03.537405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:03.537542 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:03.537956 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:03.537969 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:03.537971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:03.538310 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:03.538676 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:03.539031 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 01:17:03.539315 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":398,"threads_started":1}
W20260925 01:17:03.541070 24415 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47036
W20260925 01:17:03.542203 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 24292: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:17:03.542375 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 24292
I20260925 01:17:03.558830 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 23932
I20260925 01:17:03.560333 23994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:03.662451 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 23932
2026-09-25T01:17:03Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3696 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:17:03 dist-test-slave-s12s krb5kdc[24438](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:17:03 dist-test-slave-s12s krb5kdc[24438](info): set up 2 sockets
Sep 25 01:17:03 dist-test-slave-s12s krb5kdc[24438](info): commencing operation
krb5kdc: starting...
W20260925 01:17:04.819541 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.115s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:17:04 dist-test-slave-s12s krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:17:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:17:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:17:05.032008 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39411
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:39411
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:05.353430 24451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:05.353793 24451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:05.353981 24451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:05.361353 24451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:05.361516 24451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:05.361637 24451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:05.361753 24451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:05.361891 24451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:17:05.371726 24451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:39411
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:05.373095 24451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:05.374500 24451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:05.386940 24459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:05.387063 24456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:05.386970 24457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:05.387728 24451 server_base.cc:1034] running on GCE node
I20260925 01:17:05.388523 24451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:05.389991 24451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:05.391355 24451 hybrid_clock.cc:648] HybridClock initialized: now 1790299025391300 us; error 38 us; skew 500 ppm
I20260925 01:17:05.391737 24451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:05 dist-test-slave-s12s krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299025, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:05.394989 24451 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:17:05.396369 24451 webserver.cc:460] Webserver started at http://127.20.102.190:43855/ using document root <none> and password file <none>
I20260925 01:17:05.397152 24451 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:05.397253 24451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:05.397513 24451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:05.400501 24451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "ab87a80d0217405d8a91053fca022962"
format_stamp: "Formatted at 2026-09-25 01:17:05 on dist-test-slave-s12s"
server_key: "5a68259fa4c8cdc769bd7555889f3c56"
server_key_iv: "502f468567cc87bf6c2b69d6074a7301"
server_key_version: "encryptionkey@0"
I20260925 01:17:05.401283 24451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "ab87a80d0217405d8a91053fca022962"
format_stamp: "Formatted at 2026-09-25 01:17:05 on dist-test-slave-s12s"
server_key: "5a68259fa4c8cdc769bd7555889f3c56"
server_key_iv: "502f468567cc87bf6c2b69d6074a7301"
server_key_version: "encryptionkey@0"
I20260925 01:17:05.406407 24451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:17:05.410072 24466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:05.411445 24451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:17:05.411633 24451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "ab87a80d0217405d8a91053fca022962"
format_stamp: "Formatted at 2026-09-25 01:17:05 on dist-test-slave-s12s"
server_key: "5a68259fa4c8cdc769bd7555889f3c56"
server_key_iv: "502f468567cc87bf6c2b69d6074a7301"
server_key_version: "encryptionkey@0"
I20260925 01:17:05.411827 24451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:05.432539 24451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:05.436555 24451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:05.436862 24451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:05.454731 24451 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:39411
I20260925 01:17:05.454761 24517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:39411 every 8 connection(s)
I20260925 01:17:05.456075 24451 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:17:05.458388 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 24451
I20260925 01:17:05.458627 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:17:05.459333 20890 external_mini_cluster.cc:1428] Setting key 70420fb58ee2e7ed43975f7fa2b5167c
I20260925 01:17:05.459964 24518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:05.473677 24518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962: Bootstrap starting.
Sep 25 01:17:05 dist-test-slave-s12s krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:05.477178 24518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:05.478506 24518 log.cc:824] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:05.481748 24518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962: No bootstrap required, opened a new log
I20260925 01:17:05.484032 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:05.461417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.20.102.190:39411)
0925 01:17:05.462220 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:05.462235 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:05.462285 (+ 50us) server_negotiation.cc:407] Connection header received
0925 01:17:05.464631 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:05.464658 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:05.465195 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:05.465605 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:05.466911 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:05.468923 (+ 2012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:05.470228 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:05.471353 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:05.471543 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:05.475577 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:05.475588 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:05.475606 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:05.475651 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:05.479587 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:05.480281 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:05.480289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:05.480296 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:05.480493 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:05.480890 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:05.480899 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:05.480902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:05.481558 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:05.481856 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:05.482302 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 01:17:05.482757 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":306,"threads_started":1}
I20260925 01:17:05.490209 24518 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab87a80d0217405d8a91053fca022962" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39411 } }
I20260925 01:17:05.490506 24518 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:05.490600 24518 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab87a80d0217405d8a91053fca022962, State: Initialized, Role: FOLLOWER
I20260925 01:17:05.491137 24518 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab87a80d0217405d8a91053fca022962" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39411 } }
I20260925 01:17:05.491377 24518 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:17:05.491515 24518 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:17:05.491735 24518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:05.493933 24518 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab87a80d0217405d8a91053fca022962" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39411 } }
I20260925 01:17:05.494401 24518 leader_election.cc:302] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab87a80d0217405d8a91053fca022962; no voters:
I20260925 01:17:05.494956 24518 leader_election.cc:288] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:17:05.495096 24523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:05.495833 24523 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [term 1 LEADER]: Becoming Leader. State: Replica: ab87a80d0217405d8a91053fca022962, State: Running, Role: LEADER
I20260925 01:17:05.496408 24523 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab87a80d0217405d8a91053fca022962" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39411 } }
I20260925 01:17:05.497256 24518 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:05.500115 24524 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab87a80d0217405d8a91053fca022962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab87a80d0217405d8a91053fca022962" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39411 } } }
I20260925 01:17:05.500701 24524 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:05.499905 24525 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab87a80d0217405d8a91053fca022962. Latest consensus state: current_term: 1 leader_uuid: "ab87a80d0217405d8a91053fca022962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab87a80d0217405d8a91053fca022962" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39411 } } }
I20260925 01:17:05.501194 24525 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:05.505765 24532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:17:05.511471 24532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:17:05.520519 24532 catalog_manager.cc:1347] Generated new cluster ID: 2dd416dc501c4d92afc40f75f7563a5c
I20260925 01:17:05.520650 24532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:17:05.532859 24532 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:17:05.534088 24532 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:17:05.539628 24532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab87a80d0217405d8a91053fca022962: Generated new TSK 0
I20260925 01:17:05.540347 24532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:17:05.596517 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39411
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:05.923601 24544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:05.923866 24544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:05.924032 24544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:05.931701 24544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:05.931830 24544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:05.932014 24544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:17:05.941666 24544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:05.942906 24544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:05.944334 24544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:05.957248 24550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:05.957151 24552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:05.957777 24549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:05.957969 24544 server_base.cc:1034] running on GCE node
I20260925 01:17:05.958709 24544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:05.959698 24544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:05.960918 24544 hybrid_clock.cc:648] HybridClock initialized: now 1790299025960872 us; error 62 us; skew 500 ppm
I20260925 01:17:05.961239 24544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:05 dist-test-slave-s12s krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299025, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:05.964807 24544 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:17:05.966252 24544 webserver.cc:460] Webserver started at http://127.20.102.129:45653/ using document root <none> and password file <none>
I20260925 01:17:05.966912 24544 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:05.967056 24544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:05.967377 24544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:05.970287 24544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "6bb095529daa48bcba521096038dfd07"
format_stamp: "Formatted at 2026-09-25 01:17:05 on dist-test-slave-s12s"
server_key: "c1e38aad66c89a102ff5fe9053830476"
server_key_iv: "eccd9911c2abb10c4863f8d6dc5fc61e"
server_key_version: "encryptionkey@0"
I20260925 01:17:05.971086 24544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "6bb095529daa48bcba521096038dfd07"
format_stamp: "Formatted at 2026-09-25 01:17:05 on dist-test-slave-s12s"
server_key: "c1e38aad66c89a102ff5fe9053830476"
server_key_iv: "eccd9911c2abb10c4863f8d6dc5fc61e"
server_key_version: "encryptionkey@0"
I20260925 01:17:05.976152 24544 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:17:05.979624 24559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:05.981035 24544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:05.981215 24544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "6bb095529daa48bcba521096038dfd07"
format_stamp: "Formatted at 2026-09-25 01:17:05 on dist-test-slave-s12s"
server_key: "c1e38aad66c89a102ff5fe9053830476"
server_key_iv: "eccd9911c2abb10c4863f8d6dc5fc61e"
server_key_version: "encryptionkey@0"
I20260925 01:17:05.981415 24544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:05.994851 24544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:05.999439 24544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:05.999749 24544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:06.000814 24544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:06.002190 24544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:06.002262 24544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:06.002377 24544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:06.002439 24544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:06.036365 24544 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:39663
I20260925 01:17:06.036461 24671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:39663 every 8 connection(s)
I20260925 01:17:06.037670 24544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:17:06.047036 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 24544
I20260925 01:17:06.047269 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:17:06.048120 20890 external_mini_cluster.cc:1428] Setting key ebc9a0874ce2b03a05dfd4ba79a92e5c
Sep 25 01:17:06 dist-test-slave-s12s krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299025, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:06.060842 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:06.040092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:37529 (local address 127.20.102.190:39411)
0925 01:17:06.040513 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:06.040519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:06.041627 (+ 1108us) server_negotiation.cc:407] Connection header received
0925 01:17:06.043496 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:06.043507 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:06.043605 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:06.043721 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:06.045214 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:06.045906 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:06.047634 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:06.048296 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:06.048435 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:06.053329 (+ 4894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:06.053337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:06.053345 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:06.053388 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:06.057232 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:06.058159 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:06.058163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:06.058166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:06.058279 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:06.058822 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:06.058826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:06.058829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:06.059156 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:06.059297 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:06.060227 (+ 930us) server_negotiation.cc:299] Negotiation successful
0925 01:17:06.060429 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":201,"threads_started":1}
I20260925 01:17:06.062127 24675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:06.040713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39411 (local address 127.20.102.129:37529)
0925 01:17:06.041429 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:06.041470 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:06.043227 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:06.043944 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:06.043953 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:06.044423 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:06.045056 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:06.045072 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:06.046066 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:06.046070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:06.047441 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:06.047461 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:06.048595 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:06.048604 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:06.048922 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:06.049723 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:06.049770 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:06.053080 (+ 3310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:06.057361 (+ 4281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:06.057368 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:06.057381 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:06.058031 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:06.058385 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:06.058391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:06.058394 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:06.058648 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:06.059301 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:06.059312 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:06.059770 (+ 458us) client_negotiation.cc:769] Sending connection context
0925 01:17:06.060070 (+ 300us) client_negotiation.cc:240] Negotiation successful
0925 01:17:06.060647 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":261,"threads_started":1}
I20260925 01:17:06.063786 24672 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39411
I20260925 01:17:06.064152 24672 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:06.064965 24672 heartbeater.cc:511] Master 127.20.102.190:39411 requested a full tablet report, sending...
I20260925 01:17:06.067436 24483 ts_manager.cc:194] Registered new tserver with Master: 6bb095529daa48bcba521096038dfd07 (127.20.102.129:39663)
I20260925 01:17:06.069664 24483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:37529
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:17:06.096951 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39411
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:06.413774 24678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:06.414039 24678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:06.414198 24678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:06.421605 24678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:06.421741 24678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:06.421939 24678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:17:06.431524 24678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:06.432817 24678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:06.434166 24678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:06.446626 24684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:06.446722 24683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:06.446625 24686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:06.447656 24678 server_base.cc:1034] running on GCE node
I20260925 01:17:06.448307 24678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:06.449250 24678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:06.450450 24678 hybrid_clock.cc:648] HybridClock initialized: now 1790299026450411 us; error 61 us; skew 500 ppm
I20260925 01:17:06.450810 24678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:06 dist-test-slave-s12s krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299026, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:06.454061 24678 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:17:06.455562 24678 webserver.cc:460] Webserver started at http://127.20.102.130:46047/ using document root <none> and password file <none>
I20260925 01:17:06.456228 24678 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:06.456372 24678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:06.456733 24678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:06.459470 24678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "ec4cfeb9a42c408d995b9f4bb82de887"
format_stamp: "Formatted at 2026-09-25 01:17:06 on dist-test-slave-s12s"
server_key: "d18a9f68c72c6447feb3e514601daad4"
server_key_iv: "df584ded2952ff3df32c61c74b7cd988"
server_key_version: "encryptionkey@0"
I20260925 01:17:06.460307 24678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "ec4cfeb9a42c408d995b9f4bb82de887"
format_stamp: "Formatted at 2026-09-25 01:17:06 on dist-test-slave-s12s"
server_key: "d18a9f68c72c6447feb3e514601daad4"
server_key_iv: "df584ded2952ff3df32c61c74b7cd988"
server_key_version: "encryptionkey@0"
I20260925 01:17:06.465415 24678 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 01:17:06.468822 24693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:06.470155 24678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:17:06.470326 24678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "ec4cfeb9a42c408d995b9f4bb82de887"
format_stamp: "Formatted at 2026-09-25 01:17:06 on dist-test-slave-s12s"
server_key: "d18a9f68c72c6447feb3e514601daad4"
server_key_iv: "df584ded2952ff3df32c61c74b7cd988"
server_key_version: "encryptionkey@0"
I20260925 01:17:06.470559 24678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:06.503029 24678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:06.507680 24678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:06.507980 24678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:06.508967 24678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:06.510342 24678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:06.510448 24678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:06.510610 24678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:06.510682 24678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:06.543885 24678 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:43591
I20260925 01:17:06.543936 24805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:43591 every 8 connection(s)
I20260925 01:17:06.545300 24678 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:17:06.555230 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 24678
I20260925 01:17:06.555425 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:17:06.555969 20890 external_mini_cluster.cc:1428] Setting key fba0b542ed064e6dd499cf3e4a3780fe
Sep 25 01:17:06 dist-test-slave-s12s krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299026, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:06.566469 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:06.547854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:52661 (local address 127.20.102.190:39411)
0925 01:17:06.548023 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:06.548027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:06.549141 (+ 1114us) server_negotiation.cc:407] Connection header received
0925 01:17:06.550942 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:06.550948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:06.551042 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:06.551152 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:06.552609 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:06.553358 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:06.554956 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:06.555629 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:06.555757 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:06.559449 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:06.559458 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:06.559466 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:06.559513 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:06.563219 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:06.563918 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:06.563926 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:06.563932 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:06.564098 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:06.564500 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:06.564507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:06.564512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:06.564952 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:06.565111 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:06.565770 (+ 659us) server_negotiation.cc:299] Negotiation successful
0925 01:17:06.566029 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:17:06.567122 24808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:06.548255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39411 (local address 127.20.102.130:52661)
0925 01:17:06.548971 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:06.549014 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:06.550680 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:06.551385 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:06.551393 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:06.551878 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:06.552435 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:06.552452 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:06.553519 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:06.553524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:06.554742 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:06.554754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:06.555834 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:06.555840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:06.556100 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:06.556808 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:06.556829 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:06.559215 (+ 2386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:06.563361 (+ 4146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:06.563368 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:06.563380 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:06.563798 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:06.564209 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:06.564212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:06.564214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:06.564387 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:06.565075 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:06.565081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:06.565384 (+ 303us) client_negotiation.cc:769] Sending connection context
0925 01:17:06.565639 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 01:17:06.566019 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":275,"threads_started":1}
I20260925 01:17:06.568607 24806 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39411
I20260925 01:17:06.568943 24806 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:06.569675 24806 heartbeater.cc:511] Master 127.20.102.190:39411 requested a full tablet report, sending...
I20260925 01:17:06.571130 24483 ts_manager.cc:194] Registered new tserver with Master: ec4cfeb9a42c408d995b9f4bb82de887 (127.20.102.130:43591)
I20260925 01:17:06.571920 24483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:52661
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:17:06.599661 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39411
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:06.924081 24811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:06.924335 24811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:06.924427 24811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:06.931353 24811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:06.931461 24811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:06.931569 24811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:17:06.940586 24811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:06.941856 24811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:06.943143 24811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:06.959304 24816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:06.959376 24819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:06.959405 24817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:06.960214 24811 server_base.cc:1034] running on GCE node
I20260925 01:17:06.960949 24811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:06.961901 24811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:06.963135 24811 hybrid_clock.cc:648] HybridClock initialized: now 1790299026963099 us; error 49 us; skew 500 ppm
I20260925 01:17:06.963441 24811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:06 dist-test-slave-s12s krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299026, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:06.966715 24811 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:17:06.968211 24811 webserver.cc:460] Webserver started at http://127.20.102.131:41549/ using document root <none> and password file <none>
I20260925 01:17:06.968997 24811 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:06.969142 24811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:06.969462 24811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:06.972290 24811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "0f99a04bac5349bbbbc3b2c102317fb5"
format_stamp: "Formatted at 2026-09-25 01:17:06 on dist-test-slave-s12s"
server_key: "1cfe33175583fb5dc5295b77c6823a87"
server_key_iv: "ef9ed1a3f434c22500786d4d37bdb307"
server_key_version: "encryptionkey@0"
I20260925 01:17:06.973109 24811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "0f99a04bac5349bbbbc3b2c102317fb5"
format_stamp: "Formatted at 2026-09-25 01:17:06 on dist-test-slave-s12s"
server_key: "1cfe33175583fb5dc5295b77c6823a87"
server_key_iv: "ef9ed1a3f434c22500786d4d37bdb307"
server_key_version: "encryptionkey@0"
I20260925 01:17:06.978338 24811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:17:06.981789 24826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:06.983111 24811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:17:06.983292 24811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "0f99a04bac5349bbbbc3b2c102317fb5"
format_stamp: "Formatted at 2026-09-25 01:17:06 on dist-test-slave-s12s"
server_key: "1cfe33175583fb5dc5295b77c6823a87"
server_key_iv: "ef9ed1a3f434c22500786d4d37bdb307"
server_key_version: "encryptionkey@0"
I20260925 01:17:06.983502 24811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:06.996490 24811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:07.000622 24811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:07.000934 24811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:07.001834 24811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:07.003259 24811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:07.003366 24811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:07.003517 24811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:07.003597 24811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:07.039682 24811 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:34017
I20260925 01:17:07.039738 24938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:34017 every 8 connection(s)
I20260925 01:17:07.041442 24811 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:17:07.047911 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 24811
I20260925 01:17:07.048138 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:17:07.048738 20890 external_mini_cluster.cc:1428] Setting key 36d4193d7fa9d177ef03715deca810ad
Sep 25 01:17:07 dist-test-slave-s12s krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299026, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:07.062511 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:07.044199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:53449 (local address 127.20.102.190:39411)
0925 01:17:07.044404 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:07.044410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:07.045609 (+ 1199us) server_negotiation.cc:407] Connection header received
0925 01:17:07.047559 (+ 1950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:07.047563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:07.047633 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:07.047720 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:07.049801 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:07.050936 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:07.052558 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:07.053075 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:07.053245 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:07.056757 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:07.056763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:07.056766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:07.056801 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:07.059369 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:07.060122 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:07.060129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:07.060132 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:07.060270 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:07.060717 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:07.060722 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:07.060725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:07.061128 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:07.061283 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:07.061915 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 01:17:07.062130 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 01:17:07.063766 24941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:07.044667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39411 (local address 127.20.102.131:53449)
0925 01:17:07.045446 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:07.045499 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:07.047245 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:07.048025 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:07.048039 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:07.048787 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:07.049621 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:07.049640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:07.051095 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:07.051102 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:07.052381 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:07.052393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:07.053331 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:07.053341 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:07.053607 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:07.054252 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:07.054276 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:07.056515 (+ 2239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:07.059571 (+ 3056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:07.059579 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:07.059596 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:07.060002 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:07.060384 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:07.060389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:07.060393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:07.060577 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:07.061253 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:07.061261 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:07.061567 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 01:17:07.061827 (+ 260us) client_negotiation.cc:240] Negotiation successful
0925 01:17:07.062337 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":265,"threads_started":1}
I20260925 01:17:07.065102 24939 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39411
I20260925 01:17:07.065408 24939 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:07.066114 24939 heartbeater.cc:511] Master 127.20.102.190:39411 requested a full tablet report, sending...
I20260925 01:17:07.067482 24483 ts_manager.cc:194] Registered new tserver with Master: 0f99a04bac5349bbbbc3b2c102317fb5 (127.20.102.131:34017)
I20260925 01:17:07.068264 24483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:53449
I20260925 01:17:07.072759 24672 heartbeater.cc:503] Master 127.20.102.190:39411 was elected leader, sending a full tablet report...
I20260925 01:17:07.076277 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:17:07 dist-test-slave-s12s krb5kdc[24438](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 01:17:07 dist-test-slave-s12s krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:17:07.123880 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:07.109378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49470 (local address 127.20.102.129:39663)
0925 01:17:07.109964 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:07.109970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:07.109996 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:17:07.110058 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:07.110064 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:07.110275 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:07.110402 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:07.111825 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:07.112885 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:07.114044 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:07.114590 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:07.114683 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:07.118462 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:07.118473 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:07.118486 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:07.118523 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:07.121445 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:07.122072 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:07.122079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:07.122081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:07.122195 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:07.122523 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:07.122526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:07.122528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:07.122853 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:07.123077 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:07.123290 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 01:17:07.123505 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":294,"threads_started":1}
Sep 25 01:17:07 dist-test-slave-s12s krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:07.140575 24674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:07.127901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54056 (local address 127.20.102.190:39411)
0925 01:17:07.128126 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:07.128131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:07.128266 (+ 135us) server_negotiation.cc:407] Connection header received
0925 01:17:07.128396 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:07.128400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:07.128462 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:07.128553 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:07.129732 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:07.130423 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:07.131512 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:07.131934 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:07.132023 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:07.135629 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:07.135634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:07.135638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:07.135670 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:07.138244 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:07.138825 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:07.138828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:07.138830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:07.138974 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:07.139268 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:07.139271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:07.139273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:07.139579 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:07.139759 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:07.140092 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 01:17:07.140246 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260925 01:17:07.141222 24483 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54056
I20260925 01:17:07.142311 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 24544
I20260925 01:17:07.154822 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:07.143240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49480 (local address 127.20.102.129:39663)
0925 01:17:07.143413 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:07.143417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:07.143635 (+ 218us) server_negotiation.cc:407] Connection header received
0925 01:17:07.143758 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:07.143762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:07.143826 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:07.143913 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:07.145078 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:07.145756 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:07.146958 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:07.147389 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:07.147491 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:07.149651 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:07.149656 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:07.149659 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:07.149692 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:07.152188 (+ 2496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:07.152865 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:07.152871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:07.152874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:07.153033 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:07.153364 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:07.153367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:07.153369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:07.153644 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:07.153909 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:07.154251 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 01:17:07.154473 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:17:07.155483 24667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:07.238341 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 24544
I20260925 01:17:07.255944 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 24678
Sep 25 01:17:07 dist-test-slave-s12s krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:17:07.272091 24951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:07.256928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34342 (local address 127.20.102.130:43591)
0925 01:17:07.257531 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:07.257538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:07.257565 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:17:07.257632 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:07.257638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:07.257854 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:07.257997 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:07.259153 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:07.260182 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:07.261631 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:07.262259 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:07.262369 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:07.266408 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:07.266451 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:07.266466 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:07.266501 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:07.269464 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:07.270082 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:07.270090 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:07.270092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:07.270214 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:07.270680 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:07.270685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:07.270687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:07.271049 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:07.271278 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:07.271560 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 01:17:07.271768 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":303,"threads_started":1}
I20260925 01:17:07.273524 24801 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:07.356551 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 24678
I20260925 01:17:07.373034 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 24811
Sep 25 01:17:07 dist-test-slave-s12s krb5kdc[24438](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:17:07.389213 24953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:07.374094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58178 (local address 127.20.102.131:34017)
0925 01:17:07.374817 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:07.374823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:07.374852 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:17:07.374909 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:07.374916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:07.375121 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:07.375261 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:07.376426 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:07.377573 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:07.378820 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:07.379342 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:07.379432 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:07.383561 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:07.383572 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:07.383585 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:07.383615 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:07.386701 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:07.387293 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:07.387304 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:07.387307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:07.387427 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:07.387740 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:07.387745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:07.387748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:07.388123 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:07.388342 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:07.388600 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 01:17:07.388839 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":412,"threads_started":1}
I20260925 01:17:07.390408 24934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:07.472187 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 24811
I20260925 01:17:07.488195 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 24451
I20260925 01:17:07.489359 24513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:07.593416 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 24451
2026-09-25T01:17:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3931 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:17:07 dist-test-slave-s12s krb5kdc[24958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:17:07 dist-test-slave-s12s krb5kdc[24958](info): set up 2 sockets
Sep 25 01:17:07 dist-test-slave-s12s krb5kdc[24958](info): commencing operation
krb5kdc: starting...
W20260925 01:17:08.689700 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:17:08 dist-test-slave-s12s krb5kdc[24958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:17:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:17:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:17:08.885356 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:45197
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:45197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:09.218048 24971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:09.218398 24971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:09.218561 24971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:09.225804 24971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:09.225946 24971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:09.226052 24971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:09.226156 24971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:09.226245 24971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:17:09.236029 24971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:45197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:45197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:09.237357 24971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:09.238675 24971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:09.250038 24976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:09.250157 24979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:09.250473 24977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:09.251365 24971 server_base.cc:1034] running on GCE node
I20260925 01:17:09.252146 24971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:09.253954 24971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:09.255188 24971 hybrid_clock.cc:648] HybridClock initialized: now 1790299029255137 us; error 43 us; skew 500 ppm
I20260925 01:17:09.255535 24971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:09 dist-test-slave-s12s krb5kdc[24958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299029, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:09.258505 24971 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:17:09.259629 24971 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:09.259769 24971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:09.260125 24971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:09.263038 24971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "52e15b8cf2b74ed9be24f4fb35414ad7"
format_stamp: "Formatted at 2026-09-25 01:17:09 on dist-test-slave-s12s"
server_key: "90b9d52a8d0dba074bd0b718793a91ea"
server_key_iv: "57c3f063daab6617b077fe18ad2a8984"
server_key_version: "encryptionkey@0"
I20260925 01:17:09.263809 24971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "52e15b8cf2b74ed9be24f4fb35414ad7"
format_stamp: "Formatted at 2026-09-25 01:17:09 on dist-test-slave-s12s"
server_key: "90b9d52a8d0dba074bd0b718793a91ea"
server_key_iv: "57c3f063daab6617b077fe18ad2a8984"
server_key_version: "encryptionkey@0"
I20260925 01:17:09.268762 24971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:17:09.272079 24985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:09.273377 24971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:09.273567 24971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "52e15b8cf2b74ed9be24f4fb35414ad7"
format_stamp: "Formatted at 2026-09-25 01:17:09 on dist-test-slave-s12s"
server_key: "90b9d52a8d0dba074bd0b718793a91ea"
server_key_iv: "57c3f063daab6617b077fe18ad2a8984"
server_key_version: "encryptionkey@0"
I20260925 01:17:09.273783 24971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:09.284303 24971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:09.287967 24971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:09.288231 24971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:09.305898 24971 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:45197
I20260925 01:17:09.305918 25036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:45197 every 8 connection(s)
I20260925 01:17:09.306766 24971 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:17:09.310361 25037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:09.312129 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 24971
I20260925 01:17:09.312363 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:17:09.313172 20890 external_mini_cluster.cc:1428] Setting key ba93ff00a727902d61fa9d325310bbc0
I20260925 01:17:09.323210 25037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7: Bootstrap starting.
Sep 25 01:17:09 dist-test-slave-s12s krb5kdc[24958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:09.326522 25037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:09.327795 25037 log.cc:824] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:09.330627 25037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7: No bootstrap required, opened a new log
I20260925 01:17:09.335100 25040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:09.315321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.20.102.190:45197)
0925 01:17:09.316091 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:09.316102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:09.316142 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:17:09.317755 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:09.317782 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:09.318144 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:09.318494 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:09.319944 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:09.321347 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:09.322569 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:09.323220 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:09.323409 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:09.327744 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:09.327751 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:09.327765 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:09.327801 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:09.331106 (+ 3305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:09.331902 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:09.331908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:09.331914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:09.332067 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:09.332477 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:09.332480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:09.332482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:09.333086 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:09.333313 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:09.333764 (+ 451us) server_negotiation.cc:299] Negotiation successful
0925 01:17:09.334114 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":339,"threads_started":1}
I20260925 01:17:09.339517 25037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e15b8cf2b74ed9be24f4fb35414ad7" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45197 } }
I20260925 01:17:09.339850 25037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:09.339970 25037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e15b8cf2b74ed9be24f4fb35414ad7, State: Initialized, Role: FOLLOWER
I20260925 01:17:09.340497 25037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e15b8cf2b74ed9be24f4fb35414ad7" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45197 } }
I20260925 01:17:09.340752 25037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:17:09.340907 25037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:17:09.341086 25037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:09.343143 25037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e15b8cf2b74ed9be24f4fb35414ad7" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45197 } }
I20260925 01:17:09.343609 25037 leader_election.cc:302] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52e15b8cf2b74ed9be24f4fb35414ad7; no voters:
I20260925 01:17:09.344233 25037 leader_election.cc:288] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:17:09.344374 25042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:09.345243 25042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [term 1 LEADER]: Becoming Leader. State: Replica: 52e15b8cf2b74ed9be24f4fb35414ad7, State: Running, Role: LEADER
I20260925 01:17:09.345744 25042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e15b8cf2b74ed9be24f4fb35414ad7" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45197 } }
I20260925 01:17:09.346415 25037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:09.348850 25044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52e15b8cf2b74ed9be24f4fb35414ad7. Latest consensus state: current_term: 1 leader_uuid: "52e15b8cf2b74ed9be24f4fb35414ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e15b8cf2b74ed9be24f4fb35414ad7" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45197 } } }
I20260925 01:17:09.348906 25043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52e15b8cf2b74ed9be24f4fb35414ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e15b8cf2b74ed9be24f4fb35414ad7" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45197 } } }
I20260925 01:17:09.349262 25044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:09.349323 25043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:09.351305 25050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:17:09.356442 25050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:17:09.365119 25050 catalog_manager.cc:1347] Generated new cluster ID: 8abf520f40fb46bdbf642f3db4554c62
I20260925 01:17:09.365239 25050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:17:09.382179 25050 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:17:09.383853 25050 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:17:09.391969 25050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7: Generated new TSK 0
I20260925 01:17:09.392935 25050 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:17:09.448668 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:45197
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:09.775069 25063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:09.775357 25063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:09.775456 25063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:09.782804 25063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:09.782898 25063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:09.783012 25063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:17:09.792762 25063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:45197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:09.793973 25063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:09.795213 25063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:09.808357 25068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:09.808404 25069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:09.808434 25071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:09.809329 25063 server_base.cc:1034] running on GCE node
I20260925 01:17:09.809993 25063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:09.810900 25063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:09.812135 25063 hybrid_clock.cc:648] HybridClock initialized: now 1790299029812085 us; error 37 us; skew 500 ppm
I20260925 01:17:09.812466 25063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:09 dist-test-slave-s12s krb5kdc[24958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299029, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:09.825313 25063 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:17:09.826617 25063 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:09.826706 25063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:09.827175 25063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:09.830461 25063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "32d861bb1b6c48778f1cc3f5fb601bd6"
format_stamp: "Formatted at 2026-09-25 01:17:09 on dist-test-slave-s12s"
server_key: "acb7bad1cd7b0708bafacb5b54b2308b"
server_key_iv: "c5208c786cc4e86ca3fdcad8a0f21d1a"
server_key_version: "encryptionkey@0"
I20260925 01:17:09.831274 25063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "32d861bb1b6c48778f1cc3f5fb601bd6"
format_stamp: "Formatted at 2026-09-25 01:17:09 on dist-test-slave-s12s"
server_key: "acb7bad1cd7b0708bafacb5b54b2308b"
server_key_iv: "c5208c786cc4e86ca3fdcad8a0f21d1a"
server_key_version: "encryptionkey@0"
I20260925 01:17:09.836520 25063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260925 01:17:09.839742 25077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:09.841061 25063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:09.841228 25063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "32d861bb1b6c48778f1cc3f5fb601bd6"
format_stamp: "Formatted at 2026-09-25 01:17:09 on dist-test-slave-s12s"
server_key: "acb7bad1cd7b0708bafacb5b54b2308b"
server_key_iv: "c5208c786cc4e86ca3fdcad8a0f21d1a"
server_key_version: "encryptionkey@0"
I20260925 01:17:09.841434 25063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:09.860814 25063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:09.864975 25063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:09.865244 25063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:09.866139 25063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:09.867408 25063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:09.867530 25063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:09.867697 25063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:09.867775 25063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:09.900879 25063 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:34617
I20260925 01:17:09.900924 25189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:34617 every 8 connection(s)
I20260925 01:17:09.901698 25063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:17:09.907867 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 25063
I20260925 01:17:09.908087 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:17:09.908845 20890 external_mini_cluster.cc:1428] Setting key 869d90fbe7512d2290d0e1717e981aa1
Sep 25 01:17:09 dist-test-slave-s12s krb5kdc[24958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299029, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:09.921461 25192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:09.904095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:45425 (local address 127.20.102.190:45197)
0925 01:17:09.904530 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:09.904537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:09.905604 (+ 1067us) server_negotiation.cc:407] Connection header received
0925 01:17:09.907279 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:09.907284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:09.907355 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:09.907476 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:09.909184 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:09.910061 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:09.911679 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:09.912452 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:09.912597 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:09.915889 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:09.915897 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:09.915905 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:09.915948 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:09.918497 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:09.919214 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:09.919224 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:09.919227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:09.919339 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:09.919755 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:09.919760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:09.919763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:09.920026 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:09.920178 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:09.920889 (+ 711us) server_negotiation.cc:299] Negotiation successful
0925 01:17:09.921114 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":205,"threads_started":1}
I20260925 01:17:09.922362 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:09.904742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:45197 (local address 127.20.102.129:45425)
0925 01:17:09.905438 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:09.905480 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:09.907028 (+ 1548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:09.907638 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:09.907645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:09.908260 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:09.909013 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:09.909033 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:09.910200 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:09.910205 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:09.911526 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:09.911539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:09.912657 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:09.912661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:09.912907 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:09.913577 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:09.913599 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:09.915668 (+ 2069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:09.918644 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:09.918653 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:09.918667 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:09.919085 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:09.919449 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:09.919454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:09.919457 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:09.919634 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:09.920158 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:09.920166 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:09.920475 (+ 309us) client_negotiation.cc:769] Sending connection context
0925 01:17:09.920777 (+ 302us) client_negotiation.cc:240] Negotiation successful
0925 01:17:09.921269 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":276,"threads_started":1}
I20260925 01:17:09.923873 25190 heartbeater.cc:347] Connected to a master server at 127.20.102.190:45197
I20260925 01:17:09.924192 25190 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:09.924901 25190 heartbeater.cc:511] Master 127.20.102.190:45197 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
I20260925 01:17:09.926868 25002 ts_manager.cc:194] Registered new tserver with Master: 32d861bb1b6c48778f1cc3f5fb601bd6 (127.20.102.129:34617)
I20260925 01:17:09.928274 25002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:45425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:17:09.950145 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:45197
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:10.267640 25196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:10.267911 25196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:10.268008 25196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:10.275450 25196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:10.275564 25196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:10.275682 25196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:17:10.285586 25196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:45197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:10.286898 25196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:10.288163 25196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:10.302615 25202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:10.303031 25201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:10.303592 25204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:10.303887 25196 server_base.cc:1034] running on GCE node
I20260925 01:17:10.304553 25196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:10.305629 25196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:10.306854 25196 hybrid_clock.cc:648] HybridClock initialized: now 1790299030306812 us; error 40 us; skew 500 ppm
I20260925 01:17:10.307183 25196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:10 dist-test-slave-s12s krb5kdc[24958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:10.310436 25196 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:17:10.311637 25196 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:10.311724 25196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:10.312075 25196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:10.315100 25196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "2e29b7d0db944f7897367a8c6b9ad732"
format_stamp: "Formatted at 2026-09-25 01:17:10 on dist-test-slave-s12s"
server_key: "5d0d1cea469a913d76965de0c31a4aa9"
server_key_iv: "e9933d6deeead7dab6fe664b267c1e4a"
server_key_version: "encryptionkey@0"
I20260925 01:17:10.315917 25196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "2e29b7d0db944f7897367a8c6b9ad732"
format_stamp: "Formatted at 2026-09-25 01:17:10 on dist-test-slave-s12s"
server_key: "5d0d1cea469a913d76965de0c31a4aa9"
server_key_iv: "e9933d6deeead7dab6fe664b267c1e4a"
server_key_version: "encryptionkey@0"
I20260925 01:17:10.321056 25196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 01:17:10.324366 25210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:10.325695 25196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:10.325857 25196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "2e29b7d0db944f7897367a8c6b9ad732"
format_stamp: "Formatted at 2026-09-25 01:17:10 on dist-test-slave-s12s"
server_key: "5d0d1cea469a913d76965de0c31a4aa9"
server_key_iv: "e9933d6deeead7dab6fe664b267c1e4a"
server_key_version: "encryptionkey@0"
I20260925 01:17:10.326083 25196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:10.351128 25196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:10.355413 25196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:10.355700 25196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:10.356613 25196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:10.357950 25196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:10.358035 25196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:10.358121 25196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:10.358175 25196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:10.390906 25196 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:36251
I20260925 01:17:10.391026 25323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:36251 every 8 connection(s)
I20260925 01:17:10.391781 25196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:17:10.397895 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 25196
I20260925 01:17:10.398110 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:17:10.398552 20890 external_mini_cluster.cc:1428] Setting key 772736c06cb0bb175cbc77cae9306083
Sep 25 01:17:10 dist-test-slave-s12s krb5kdc[24958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:10.413925 25192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:10.394380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:46569 (local address 127.20.102.190:45197)
0925 01:17:10.394557 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:10.394562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:10.395738 (+ 1176us) server_negotiation.cc:407] Connection header received
0925 01:17:10.397738 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:10.397744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:10.397839 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:10.397949 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:10.399632 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:10.400436 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:10.402002 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:10.402417 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:10.402553 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:10.406448 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:10.406457 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:10.406462 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:10.406503 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:10.410434 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:10.411171 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:10.411180 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:10.411187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:10.411363 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:10.411800 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:10.411809 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:10.411815 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:10.412221 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:10.412384 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:10.413119 (+ 735us) server_negotiation.cc:299] Negotiation successful
0925 01:17:10.413378 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:17:10.414470 25326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:10.394862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:45197 (local address 127.20.102.130:46569)
0925 01:17:10.395579 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:10.395617 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:10.397441 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:10.398176 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:10.398207 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:10.398796 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:10.399431 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:10.399454 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:10.400581 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:10.400588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:10.401855 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:10.401867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:10.402636 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:10.402643 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:10.402874 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:10.403601 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:10.403623 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:10.406212 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:10.410589 (+ 4377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:10.410596 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:10.410608 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:10.411026 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:10.411477 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:10.411481 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:10.411483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:10.411653 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:10.412346 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:10.412352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:10.412652 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 01:17:10.412932 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 01:17:10.413307 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":299,"threads_started":1}
I20260925 01:17:10.416035 25324 heartbeater.cc:347] Connected to a master server at 127.20.102.190:45197
I20260925 01:17:10.416409 25324 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:10.417349 25324 heartbeater.cc:511] Master 127.20.102.190:45197 requested a full tablet report, sending...
I20260925 01:17:10.419041 25002 ts_manager.cc:194] Registered new tserver with Master: 2e29b7d0db944f7897367a8c6b9ad732 (127.20.102.130:36251)
I20260925 01:17:10.419888 25002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:46569
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:17:10.446576 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:45197
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:10.770694 25329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:10.771031 25329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:10.771217 25329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:10.778746 25329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:10.778895 25329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:10.779079 25329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:17:10.788960 25329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:45197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:10.790338 25329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:10.791729 25329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:10.804704 25334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:10.804656 25335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:10.804618 25337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:10.806025 25329 server_base.cc:1034] running on GCE node
I20260925 01:17:10.806535 25329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:10.807375 25329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:10.808614 25329 hybrid_clock.cc:648] HybridClock initialized: now 1790299030808563 us; error 56 us; skew 500 ppm
I20260925 01:17:10.808941 25329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:10 dist-test-slave-s12s krb5kdc[24958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:10.812180 25329 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:17:10.813372 25329 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:10.813462 25329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:10.813711 25329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:10.816500 25329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "ae089bf525c4430d91307fad175200e5"
format_stamp: "Formatted at 2026-09-25 01:17:10 on dist-test-slave-s12s"
server_key: "092bba3634af2c970d2891eab42ed3c3"
server_key_iv: "69c0faf7574246133f8bd192d70adbd9"
server_key_version: "encryptionkey@0"
I20260925 01:17:10.817303 25329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "ae089bf525c4430d91307fad175200e5"
format_stamp: "Formatted at 2026-09-25 01:17:10 on dist-test-slave-s12s"
server_key: "092bba3634af2c970d2891eab42ed3c3"
server_key_iv: "69c0faf7574246133f8bd192d70adbd9"
server_key_version: "encryptionkey@0"
I20260925 01:17:10.822181 25329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 01:17:10.825520 25343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:10.826756 25329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:17:10.826884 25329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "ae089bf525c4430d91307fad175200e5"
format_stamp: "Formatted at 2026-09-25 01:17:10 on dist-test-slave-s12s"
server_key: "092bba3634af2c970d2891eab42ed3c3"
server_key_iv: "69c0faf7574246133f8bd192d70adbd9"
server_key_version: "encryptionkey@0"
I20260925 01:17:10.827065 25329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:10.881875 25329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:10.886368 25329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:10.886600 25329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:10.887513 25329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:10.888857 25329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:10.888945 25329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:10.889046 25329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:10.889101 25329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:10.923172 25329 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:33925
I20260925 01:17:10.923285 25455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:33925 every 8 connection(s)
I20260925 01:17:10.924070 25329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:17:10.931075 25190 heartbeater.cc:503] Master 127.20.102.190:45197 was elected leader, sending a full tablet report...
I20260925 01:17:10.934391 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 25329
I20260925 01:17:10.934588 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:17:10.935110 20890 external_mini_cluster.cc:1428] Setting key 2301901c1e8506bd2702bbc09e04f9e9
Sep 25 01:17:10 dist-test-slave-s12s krb5kdc[24958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299030, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:10.945641 25458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:10.926722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:36031 (local address 127.20.102.190:45197)
0925 01:17:10.927222 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:10.927230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:10.928222 (+ 992us) server_negotiation.cc:407] Connection header received
0925 01:17:10.930060 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:10.930065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:10.930152 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:10.930250 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:10.932224 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:10.933370 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:10.935490 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:10.935930 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:10.936030 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:10.939825 (+ 3795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:10.939830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:10.939833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:10.939864 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:10.942261 (+ 2397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:10.942994 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:10.943002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:10.943009 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:10.943129 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:10.943543 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:10.943549 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:10.943556 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:10.943861 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:10.944031 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:10.945035 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0925 01:17:10.945260 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":230,"threads_started":1}
I20260925 01:17:10.946954 25459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:10.927272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:45197 (local address 127.20.102.131:36031)
0925 01:17:10.928053 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:10.928091 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:10.929781 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:10.930605 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:10.930615 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:10.931288 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:10.932026 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:10.932045 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:10.933505 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:10.933508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:10.935345 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:10.935360 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:10.936148 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:10.936154 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:10.936394 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:10.937232 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:10.937267 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:10.939629 (+ 2362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:10.942419 (+ 2790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:10.942427 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:10.942442 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:10.942866 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:10.943239 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:10.943244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:10.943247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:10.943431 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:10.944015 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:10.944024 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:10.944547 (+ 523us) client_negotiation.cc:769] Sending connection context
0925 01:17:10.944911 (+ 364us) client_negotiation.cc:240] Negotiation successful
0925 01:17:10.945421 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":313,"threads_started":1}
I20260925 01:17:10.948457 25456 heartbeater.cc:347] Connected to a master server at 127.20.102.190:45197
I20260925 01:17:10.948807 25456 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:10.949479 25456 heartbeater.cc:511] Master 127.20.102.190:45197 requested a full tablet report, sending...
I20260925 01:17:10.951148 25001 ts_manager.cc:194] Registered new tserver with Master: ae089bf525c4430d91307fad175200e5 (127.20.102.131:33925)
I20260925 01:17:10.952384 25001 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:36031
I20260925 01:17:10.954344 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:17:10.971473 25458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:10.958913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37928 (local address 127.20.102.190:45197)
0925 01:17:10.959080 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:10.959085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:10.959544 (+ 459us) server_negotiation.cc:407] Connection header received
0925 01:17:10.959683 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:10.959687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:10.959743 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:10.959820 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:10.961291 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:10.962002 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:10.963219 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:10.963607 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:10.963692 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:10.965868 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:10.965873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:10.965876 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:10.965905 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:10.969115 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:10.969739 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:10.969743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:10.969745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:10.969855 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:10.970183 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:10.970191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:10.970197 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:10.970527 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:10.970700 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:10.971100 (+ 400us) server_negotiation.cc:299] Negotiation successful
0925 01:17:10.971257 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 01:17:10.975425 25002 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:17:10.982378 25002 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:17:11.020576 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:10.994936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:33925 (local address 127.0.0.1:53724)
0925 01:17:11.010976 (+ 16040us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:11.011002 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:11.011154 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:11.013928 (+ 2774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:11.013935 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:11.013966 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:11.014360 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:11.014374 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.016222 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.016227 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:11.018332 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:11.018343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.019574 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.019582 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:11.019777 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:11.019839 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 01:17:11.020006 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 01:17:11.020201 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15871,"thread_start_us":10038,"threads_started":1}
I20260925 01:17:11.020584 25472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:10.995691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.20.102.131:33925)
0925 01:17:11.013341 (+ 17650us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:11.013349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:11.013383 (+ 34us) server_negotiation.cc:407] Connection header received
0925 01:17:11.013479 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:11.013486 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:11.013711 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:11.013913 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:11.014603 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.016065 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.018468 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.019389 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.019601 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:11.019777 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:11.020039 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 01:17:11.020204 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17447,"thread_start_us":17260,"threads_started":1}
I20260925 01:17:11.021226 25471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:10.994653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:36251 (local address 127.0.0.1:37110)
0925 01:17:11.005473 (+ 10820us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:11.005523 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:11.005746 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:11.012846 (+ 7100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:11.012856 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:11.012894 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:11.013326 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:11.013339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.015349 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.015354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:11.017758 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:11.017773 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.019976 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.019981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:11.020257 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:11.020387 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 01:17:11.020473 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 01:17:11.020625 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10469,"mutex_wait_us":146,"thread_start_us":17822,"threads_started":1}
I20260925 01:17:11.021667 25470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:10.994751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37110 (local address 127.20.102.130:36251)
0925 01:17:11.011790 (+ 17039us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:11.011803 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:11.011841 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:17:11.011964 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:11.011976 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:11.012302 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:11.013519 (+ 1217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:11.013779 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.015200 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.017916 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.019420 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.019590 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:11.019777 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:11.021164 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0925 01:17:11.021346 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16824,"thread_start_us":16666,"threads_started":1}
I20260925 01:17:11.021755 25473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:11.007386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:34617 (local address 127.0.0.1:55938)
0925 01:17:11.012131 (+ 4745us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:11.012155 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:11.012293 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:11.013556 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:11.013560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:11.013584 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:11.013999 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:11.014022 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.015562 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.015568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:11.017777 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:11.017790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.019959 (+ 2169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.019967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:11.020254 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:11.020384 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 01:17:11.020473 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 01:17:11.021036 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4584,"mutex_wait_us":585,"thread_start_us":4455,"threads_started":1}
I20260925 01:17:11.022356 25474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:11.007532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.20.102.129:34617)
0925 01:17:11.013018 (+ 5486us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:11.013026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:11.013052 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:17:11.013139 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:11.013146 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:11.013373 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:11.013571 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:11.014223 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.015413 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.017978 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.019420 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.019625 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:11.019777 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:11.020590 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 01:17:11.022027 (+ 1437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5272,"thread_start_us":5035,"threads_started":1}
I20260925 01:17:11.024839 25258 tablet_service.cc:1467] Processing CreateTablet for tablet 3d75d93a824247f38114248567504a27 (DEFAULT_TABLE table=test-table [id=bfa1a18cc65f49a9817c9f522c4aab57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:11.024554 25391 tablet_service.cc:1467] Processing CreateTablet for tablet 3d75d93a824247f38114248567504a27 (DEFAULT_TABLE table=test-table [id=bfa1a18cc65f49a9817c9f522c4aab57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:11.026182 25258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d75d93a824247f38114248567504a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:11.026369 25391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d75d93a824247f38114248567504a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:11.026937 25125 tablet_service.cc:1467] Processing CreateTablet for tablet 3d75d93a824247f38114248567504a27 (DEFAULT_TABLE table=test-table [id=bfa1a18cc65f49a9817c9f522c4aab57]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:11.028230 25125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d75d93a824247f38114248567504a27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:11.037974 25475 tablet_bootstrap.cc:492] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732: Bootstrap starting.
I20260925 01:17:11.040686 25476 tablet_bootstrap.cc:492] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5: Bootstrap starting.
I20260925 01:17:11.041767 25475 tablet_bootstrap.cc:654] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:11.042433 25477 tablet_bootstrap.cc:492] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6: Bootstrap starting.
I20260925 01:17:11.043316 25475 log.cc:824] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:11.044531 25476 tablet_bootstrap.cc:654] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:11.045994 25477 tablet_bootstrap.cc:654] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:11.046157 25476 log.cc:824] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:11.047446 25477 log.cc:824] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:11.049588 25475 tablet_bootstrap.cc:492] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732: No bootstrap required, opened a new log
I20260925 01:17:11.049724 25476 tablet_bootstrap.cc:492] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5: No bootstrap required, opened a new log
I20260925 01:17:11.049942 25475 ts_tablet_manager.cc:1397] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260925 01:17:11.050009 25476 ts_tablet_manager.cc:1397] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260925 01:17:11.050602 25477 tablet_bootstrap.cc:492] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6: No bootstrap required, opened a new log
I20260925 01:17:11.050856 25477 ts_tablet_manager.cc:1397] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.004s
I20260925 01:17:11.057680 25475 raft_consensus.cc:348] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e29b7d0db944f7897367a8c6b9ad732" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36251 } } peers { permanent_uuid: "ae089bf525c4430d91307fad175200e5" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33925 } } peers { permanent_uuid: "32d861bb1b6c48778f1cc3f5fb601bd6" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34617 } }
I20260925 01:17:11.058211 25475 raft_consensus.cc:374] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:11.058310 25475 raft_consensus.cc:729] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e29b7d0db944f7897367a8c6b9ad732, State: Initialized, Role: FOLLOWER
I20260925 01:17:11.059154 25475 consensus_queue.cc:260] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e29b7d0db944f7897367a8c6b9ad732" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36251 } } peers { permanent_uuid: "ae089bf525c4430d91307fad175200e5" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33925 } } peers { permanent_uuid: "32d861bb1b6c48778f1cc3f5fb601bd6" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34617 } }
I20260925 01:17:11.061405 25477 raft_consensus.cc:348] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e29b7d0db944f7897367a8c6b9ad732" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36251 } } peers { permanent_uuid: "ae089bf525c4430d91307fad175200e5" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33925 } } peers { permanent_uuid: "32d861bb1b6c48778f1cc3f5fb601bd6" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34617 } }
I20260925 01:17:11.061847 25477 raft_consensus.cc:374] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:11.061937 25477 raft_consensus.cc:729] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32d861bb1b6c48778f1cc3f5fb601bd6, State: Initialized, Role: FOLLOWER
I20260925 01:17:11.062577 25476 raft_consensus.cc:348] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e29b7d0db944f7897367a8c6b9ad732" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36251 } } peers { permanent_uuid: "ae089bf525c4430d91307fad175200e5" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33925 } } peers { permanent_uuid: "32d861bb1b6c48778f1cc3f5fb601bd6" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34617 } }
I20260925 01:17:11.062747 25477 consensus_queue.cc:260] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e29b7d0db944f7897367a8c6b9ad732" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36251 } } peers { permanent_uuid: "ae089bf525c4430d91307fad175200e5" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33925 } } peers { permanent_uuid: "32d861bb1b6c48778f1cc3f5fb601bd6" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34617 } }
I20260925 01:17:11.063009 25476 raft_consensus.cc:374] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:11.063099 25476 raft_consensus.cc:729] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae089bf525c4430d91307fad175200e5, State: Initialized, Role: FOLLOWER
I20260925 01:17:11.063917 25476 consensus_queue.cc:260] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e29b7d0db944f7897367a8c6b9ad732" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36251 } } peers { permanent_uuid: "ae089bf525c4430d91307fad175200e5" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33925 } } peers { permanent_uuid: "32d861bb1b6c48778f1cc3f5fb601bd6" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34617 } }
I20260925 01:17:11.067675 25477 ts_tablet_manager.cc:1428] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
I20260925 01:17:11.069065 25324 heartbeater.cc:503] Master 127.20.102.190:45197 was elected leader, sending a full tablet report...
I20260925 01:17:11.069741 25475 ts_tablet_manager.cc:1428] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732: Time spent starting tablet: real 0.020s user 0.015s sys 0.005s
I20260925 01:17:11.070864 25456 heartbeater.cc:503] Master 127.20.102.190:45197 was elected leader, sending a full tablet report...
I20260925 01:17:11.071447 25476 ts_tablet_manager.cc:1428] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5: Time spent starting tablet: real 0.021s user 0.016s sys 0.004s
W20260925 01:17:11.143787 25325 tablet.cc:2369] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:17:11.153851 25191 tablet.cc:2369] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:17:11.175686 25457 tablet.cc:2369] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:17:11.306712 25483 raft_consensus.cc:482] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:17:11.307050 25483 raft_consensus.cc:504] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e29b7d0db944f7897367a8c6b9ad732" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36251 } } peers { permanent_uuid: "ae089bf525c4430d91307fad175200e5" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33925 } } peers { permanent_uuid: "32d861bb1b6c48778f1cc3f5fb601bd6" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34617 } }
I20260925 01:17:11.308795 25483 leader_election.cc:288] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e29b7d0db944f7897367a8c6b9ad732 (127.20.102.130:36251), 32d861bb1b6c48778f1cc3f5fb601bd6 (127.20.102.129:34617)
I20260925 01:17:11.316134 25470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:11.309482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:42683 (local address 127.20.102.130:36251)
0925 01:17:11.309656 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:11.309662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:11.310262 (+ 600us) server_negotiation.cc:407] Connection header received
0925 01:17:11.310452 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:11.310457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:11.310532 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:11.310640 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:11.311228 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.312083 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.313911 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.315118 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.315304 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:11.315384 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:11.315720 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 01:17:11.315866 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 01:17:11.316179 25459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:11.309034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:34617 (local address 127.20.102.131:57993)
0925 01:17:11.309224 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:11.309245 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:11.309373 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:11.310098 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:11.310103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:11.310145 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:11.310333 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:11.310343 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.311695 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.311701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:11.313807 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:11.313822 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.315488 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.315493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:11.315622 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:11.315680 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 01:17:11.315766 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 01:17:11.315898 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":1}
I20260925 01:17:11.316228 25484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:11.309031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:36251 (local address 127.20.102.131:42683)
0925 01:17:11.310139 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:11.310158 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:11.310323 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:11.310691 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:11.310696 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:11.310725 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:11.311053 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:11.311063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.312253 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.312256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:11.313758 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:11.313769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.315342 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.315348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:11.315544 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:11.315580 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:17:11.315678 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 01:17:11.315818 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":187,"thread_start_us":614,"threads_started":1}
I20260925 01:17:11.317001 25474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:11.309512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:57993 (local address 127.20.102.129:34617)
0925 01:17:11.309735 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:11.309744 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:11.309767 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:17:11.309851 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:11.309857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:11.309940 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:11.310036 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:11.310497 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.311555 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.313980 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.315192 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.315333 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:11.315397 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:11.316585 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0925 01:17:11.316752 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 01:17:11.317406 25278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d75d93a824247f38114248567504a27" candidate_uuid: "ae089bf525c4430d91307fad175200e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e29b7d0db944f7897367a8c6b9ad732" is_pre_election: true
I20260925 01:17:11.317972 25278 raft_consensus.cc:2454] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae089bf525c4430d91307fad175200e5 in term 0.
I20260925 01:17:11.317945 25145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d75d93a824247f38114248567504a27" candidate_uuid: "ae089bf525c4430d91307fad175200e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d861bb1b6c48778f1cc3f5fb601bd6" is_pre_election: true
I20260925 01:17:11.318454 25145 raft_consensus.cc:2454] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae089bf525c4430d91307fad175200e5 in term 0.
I20260925 01:17:11.318701 25345 leader_election.cc:302] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e29b7d0db944f7897367a8c6b9ad732, ae089bf525c4430d91307fad175200e5; no voters:
I20260925 01:17:11.319322 25483 raft_consensus.cc:2781] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:17:11.319469 25483 raft_consensus.cc:482] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:17:11.319593 25483 raft_consensus.cc:3037] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:11.321815 25483 raft_consensus.cc:504] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e29b7d0db944f7897367a8c6b9ad732" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36251 } } peers { permanent_uuid: "ae089bf525c4430d91307fad175200e5" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33925 } } peers { permanent_uuid: "32d861bb1b6c48778f1cc3f5fb601bd6" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34617 } }
I20260925 01:17:11.322428 25483 leader_election.cc:288] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [CANDIDATE]: Term 1 election: Requested vote from peers 2e29b7d0db944f7897367a8c6b9ad732 (127.20.102.130:36251), 32d861bb1b6c48778f1cc3f5fb601bd6 (127.20.102.129:34617)
I20260925 01:17:11.322772 25278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d75d93a824247f38114248567504a27" candidate_uuid: "ae089bf525c4430d91307fad175200e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e29b7d0db944f7897367a8c6b9ad732"
I20260925 01:17:11.322890 25145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d75d93a824247f38114248567504a27" candidate_uuid: "ae089bf525c4430d91307fad175200e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32d861bb1b6c48778f1cc3f5fb601bd6"
I20260925 01:17:11.322973 25278 raft_consensus.cc:3037] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:11.323127 25145 raft_consensus.cc:3037] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:11.325413 25145 raft_consensus.cc:2454] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae089bf525c4430d91307fad175200e5 in term 1.
I20260925 01:17:11.325449 25278 raft_consensus.cc:2454] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae089bf525c4430d91307fad175200e5 in term 1.
I20260925 01:17:11.325847 25346 leader_election.cc:302] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32d861bb1b6c48778f1cc3f5fb601bd6, ae089bf525c4430d91307fad175200e5; no voters:
I20260925 01:17:11.326181 25483 raft_consensus.cc:2781] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:11.326560 25483 raft_consensus.cc:686] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [term 1 LEADER]: Becoming Leader. State: Replica: ae089bf525c4430d91307fad175200e5, State: Running, Role: LEADER
I20260925 01:17:11.327062 25483 consensus_queue.cc:237] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e29b7d0db944f7897367a8c6b9ad732" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36251 } } peers { permanent_uuid: "ae089bf525c4430d91307fad175200e5" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33925 } } peers { permanent_uuid: "32d861bb1b6c48778f1cc3f5fb601bd6" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34617 } }
I20260925 01:17:11.332013 25000 catalog_manager.cc:5495] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae089bf525c4430d91307fad175200e5 (127.20.102.131). New cstate: current_term: 1 leader_uuid: "ae089bf525c4430d91307fad175200e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e29b7d0db944f7897367a8c6b9ad732" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae089bf525c4430d91307fad175200e5" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32d861bb1b6c48778f1cc3f5fb601bd6" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34617 } health_report { overall_health: UNKNOWN } } }
I20260925 01:17:11.373034 25472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:11.367783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53740 (local address 127.20.102.131:33925)
0925 01:17:11.368028 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:11.368032 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:11.368053 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:17:11.368136 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:11.368140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:11.368222 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:11.368375 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:17:11.368782 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.369510 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.370829 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.371226 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.371361 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:11.371906 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:17:11.372422 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:17:11.372584 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:11.372667 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 01:17:11.372770 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 01:17:11.381127 25145 raft_consensus.cc:1261] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6 [term 1 FOLLOWER]: Refusing update from remote peer ae089bf525c4430d91307fad175200e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:11.381136 25278 raft_consensus.cc:1261] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732 [term 1 FOLLOWER]: Refusing update from remote peer ae089bf525c4430d91307fad175200e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:11.382064 25485 consensus_queue.cc:1035] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e29b7d0db944f7897367a8c6b9ad732" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 36251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:11.382419 25483 consensus_queue.cc:1035] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32d861bb1b6c48778f1cc3f5fb601bd6" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 34617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:11.391168 25488 mvcc.cc:205] Tried to move back new op lower bound from 7333064832527327232 to 7333064832319365120. Current Snapshot: MvccSnapshot[applied={T|T < 7333064832527327232}]
I20260925 01:17:11.400826 25490 mvcc.cc:205] Tried to move back new op lower bound from 7333064832527327232 to 7333064832319365120. Current Snapshot: MvccSnapshot[applied={T|T < 7333064832527327232 or (T in {7333064832527327232})}]
I20260925 01:17:11.406980 25489 mvcc.cc:205] Tried to move back new op lower bound from 7333064832527327232 to 7333064832319365120. Current Snapshot: MvccSnapshot[applied={T|T < 7333064832527327232}]
I20260925 01:17:11.413918 25002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:11.414323 25002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37928:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:11.418927 25002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52e15b8cf2b74ed9be24f4fb35414ad7: Sending DeleteTablet for 3 replicas of tablet 3d75d93a824247f38114248567504a27
I20260925 01:17:11.420378 25391 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d75d93a824247f38114248567504a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:11 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:53724
I20260925 01:17:11.420378 25125 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d75d93a824247f38114248567504a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:11 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:55938
I20260925 01:17:11.420717 25124 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d75d93a824247f38114248567504a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:11 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:55938
I20260925 01:17:11.421450 25497 tablet_replica.cc:330] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5: stopping tablet replica
I20260925 01:17:11.421386 25258 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d75d93a824247f38114248567504a27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:11 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:37110
I20260925 01:17:11.421898 25497 raft_consensus.cc:2229] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:17:11.422606 25497 raft_consensus.cc:2258] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:11.423394 25498 tablet_replica.cc:330] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6: stopping tablet replica
I20260925 01:17:11.423820 25498 raft_consensus.cc:2229] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:11.423949 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 25063
I20260925 01:17:11.424306 25498 raft_consensus.cc:2258] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:11.426400 25497 ts_tablet_manager.cc:1905] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:11.426628 25499 tablet_replica.cc:330] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732: stopping tablet replica
I20260925 01:17:11.427014 25499 raft_consensus.cc:2229] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:11.427476 25499 raft_consensus.cc:2258] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:11.429344 25498 ts_tablet_manager.cc:1905] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:11.431895 25499 ts_tablet_manager.cc:1905] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:11.436482 25498 ts_tablet_manager.cc:1918] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:11.436640 25498 log.cc:1196] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/3d75d93a824247f38114248567504a27
I20260925 01:17:11.437170 25498 ts_tablet_manager.cc:1939] T 3d75d93a824247f38114248567504a27 P 32d861bb1b6c48778f1cc3f5fb601bd6: Deleting consensus metadata
I20260925 01:17:11.438655 25499 ts_tablet_manager.cc:1918] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:11.438807 25499 log.cc:1196] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/3d75d93a824247f38114248567504a27
I20260925 01:17:11.438983 24988 catalog_manager.cc:4841] TS 32d861bb1b6c48778f1cc3f5fb601bd6 (127.20.102.129:34617): tablet 3d75d93a824247f38114248567504a27 (table test-table [id=bfa1a18cc65f49a9817c9f522c4aab57]) successfully deleted
I20260925 01:17:11.439330 25499 ts_tablet_manager.cc:1939] T 3d75d93a824247f38114248567504a27 P 2e29b7d0db944f7897367a8c6b9ad732: Deleting consensus metadata
W20260925 01:17:11.439761 24988 catalog_manager.cc:4808] TS 32d861bb1b6c48778f1cc3f5fb601bd6 (127.20.102.129:34617): delete failed for tablet 3d75d93a824247f38114248567504a27 because the tablet was not found. No further retry: Not found: Tablet not found: 3d75d93a824247f38114248567504a27
I20260925 01:17:11.440991 24987 catalog_manager.cc:4841] TS 2e29b7d0db944f7897367a8c6b9ad732 (127.20.102.130:36251): tablet 3d75d93a824247f38114248567504a27 (table test-table [id=bfa1a18cc65f49a9817c9f522c4aab57]) successfully deleted
I20260925 01:17:11.442243 25497 ts_tablet_manager.cc:1918] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:11.442425 25497 log.cc:1196] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/3d75d93a824247f38114248567504a27
I20260925 01:17:11.442991 25497 ts_tablet_manager.cc:1939] T 3d75d93a824247f38114248567504a27 P ae089bf525c4430d91307fad175200e5: Deleting consensus metadata
I20260925 01:17:11.444511 24988 catalog_manager.cc:4841] TS ae089bf525c4430d91307fad175200e5 (127.20.102.131:33925): tablet 3d75d93a824247f38114248567504a27 (table test-table [id=bfa1a18cc65f49a9817c9f522c4aab57]) successfully deleted
Sep 25 01:17:11 dist-test-slave-s12s krb5kdc[24958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:17:11.457165 25474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:11.426851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.20.102.129:34617)
0925 01:17:11.427041 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:11.427048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:11.438467 (+ 11419us) server_negotiation.cc:407] Connection header received
0925 01:17:11.440363 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:11.440369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:11.440446 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:11.440531 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:11.442747 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.443852 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.445694 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.446213 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.446327 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:11.451334 (+ 5007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:11.451346 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:11.451360 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:11.451395 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:11.454313 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:11.454973 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:11.454980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:11.454983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:11.455119 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:11.455546 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:11.455550 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:11.455552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:11.455897 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:11.456218 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:11.456537 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 01:17:11.456819 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:17:11.457826 25185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:11.547040 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 25063
I20260925 01:17:11.565140 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 25196
Sep 25 01:17:11 dist-test-slave-s12s krb5kdc[24958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:17:11.582197 25470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:11.566043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37126 (local address 127.20.102.130:36251)
0925 01:17:11.566272 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:11.566279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:11.566329 (+ 50us) server_negotiation.cc:407] Connection header received
0925 01:17:11.566398 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:11.566403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:11.566499 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:11.566606 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:11.567786 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.568953 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.570280 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.570951 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.571087 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:11.575574 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:11.575593 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:11.575615 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:11.575662 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:11.579320 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:11.579994 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:11.580004 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:11.580008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:11.580184 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:11.580534 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:11.580540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:11.580546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:11.581030 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:11.581333 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:11.581584 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 01:17:11.581800 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 01:17:11.582927 25319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:11.672006 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 25196
I20260925 01:17:11.689682 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 25329
Sep 25 01:17:11 dist-test-slave-s12s krb5kdc[24958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:17:11.706175 25472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:11.690504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.20.102.131:33925)
0925 01:17:11.690671 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:11.690675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:11.690745 (+ 70us) server_negotiation.cc:407] Connection header received
0925 01:17:11.690969 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:11.690976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:11.691071 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:11.691197 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:11.692891 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.693692 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.694930 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:11.695401 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:11.695565 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:11.699969 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:11.699981 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:11.699998 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:11.700035 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:11.703153 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:11.703851 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:11.703858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:11.703860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:11.704007 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:11.704387 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:11.704391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:11.704396 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:11.704965 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:11.705294 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:11.705593 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 01:17:11.705818 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:17:11.706890 25451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:11.827167 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 25329
I20260925 01:17:11.847160 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 24971
I20260925 01:17:11.848330 25032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:11.958154 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 24971
2026-09-25T01:17:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4364 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T01:17:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:17:12Z Disabled control of system clock
I20260925 01:17:12.025782 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39485
--webserver_interface=127.20.102.190
--webserver_port=0
--builtin_ntp_servers=127.20.102.148:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:39485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 01:17:12.353844 25508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:12.354128 25508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:12.354209 25508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:12.361436 25508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:12.361560 25508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:12.361604 25508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:12.361660 25508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:12.361725 25508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:17:12.370628 25508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:39485
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:12.371918 25508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:12.373684 25508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:12.385361 25516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:12.385313 25513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:12.385406 25514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:12.393878 25508 server_base.cc:1034] running on GCE node
I20260925 01:17:12.394775 25508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:12.396253 25508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:12.397529 25508 hybrid_clock.cc:648] HybridClock initialized: now 1790299032397487 us; error 55 us; skew 500 ppm
I20260925 01:17:12.397965 25508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:17:12.400308 25508 webserver.cc:460] Webserver started at http://127.20.102.190:32931/ using document root <none> and password file <none>
I20260925 01:17:12.401110 25508 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:12.401244 25508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:12.401605 25508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:12.404603 25508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "ff89cefabae64b528906dfbdc4b8a692"
format_stamp: "Formatted at 2026-09-25 01:17:12 on dist-test-slave-s12s"
server_key: "befec439d1c4ad2dc22e16185fe0ad15"
server_key_iv: "c581279e86c2f6a5dd1efb6c1a9ea74d"
server_key_version: "encryptionkey@0"
I20260925 01:17:12.405483 25508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "ff89cefabae64b528906dfbdc4b8a692"
format_stamp: "Formatted at 2026-09-25 01:17:12 on dist-test-slave-s12s"
server_key: "befec439d1c4ad2dc22e16185fe0ad15"
server_key_iv: "c581279e86c2f6a5dd1efb6c1a9ea74d"
server_key_version: "encryptionkey@0"
I20260925 01:17:12.410861 25508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 01:17:12.414472 25522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:12.415812 25508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:17:12.416004 25508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "ff89cefabae64b528906dfbdc4b8a692"
format_stamp: "Formatted at 2026-09-25 01:17:12 on dist-test-slave-s12s"
server_key: "befec439d1c4ad2dc22e16185fe0ad15"
server_key_iv: "c581279e86c2f6a5dd1efb6c1a9ea74d"
server_key_version: "encryptionkey@0"
I20260925 01:17:12.416255 25508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:12.422195 25508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:12.423072 25508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:12.423386 25508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:12.441953 25508 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:39485
I20260925 01:17:12.441987 25573 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:39485 every 8 connection(s)
I20260925 01:17:12.443358 25508 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:17:12.447160 25574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:12.453675 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 25508
I20260925 01:17:12.453879 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:17:12.454507 20890 external_mini_cluster.cc:1428] Setting key 94d4ee13fbee8707e8043c3275ca873f
I20260925 01:17:12.457623 25574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692: Bootstrap starting.
I20260925 01:17:12.460999 25574 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:12.462278 25574 log.cc:824] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:12.463069 25577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:12.456450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.20.102.190:39485)
0925 01:17:12.457541 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:12.457551 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:12.457591 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:17:12.459292 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:12.459314 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:12.459719 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:12.460038 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:12.461231 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:12.461237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:12.461250 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:12.461272 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:12.461332 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:12.461352 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:12.461530 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:12.461686 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 01:17:12.461967 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":443,"threads_started":1}
I20260925 01:17:12.465221 25574 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692: No bootstrap required, opened a new log
I20260925 01:17:12.472119 25574 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff89cefabae64b528906dfbdc4b8a692" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39485 } }
I20260925 01:17:12.472469 25574 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:12.472568 25574 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff89cefabae64b528906dfbdc4b8a692, State: Initialized, Role: FOLLOWER
I20260925 01:17:12.473150 25574 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff89cefabae64b528906dfbdc4b8a692" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39485 } }
I20260925 01:17:12.473384 25574 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:17:12.473513 25574 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:17:12.473691 25574 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:12.475859 25574 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff89cefabae64b528906dfbdc4b8a692" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39485 } }
I20260925 01:17:12.476356 25574 leader_election.cc:302] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff89cefabae64b528906dfbdc4b8a692; no voters:
I20260925 01:17:12.477015 25574 leader_election.cc:288] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:17:12.477162 25579 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:12.478749 25579 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [term 1 LEADER]: Becoming Leader. State: Replica: ff89cefabae64b528906dfbdc4b8a692, State: Running, Role: LEADER
I20260925 01:17:12.479328 25574 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:12.479300 25579 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff89cefabae64b528906dfbdc4b8a692" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39485 } }
I20260925 01:17:12.482316 25581 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff89cefabae64b528906dfbdc4b8a692. Latest consensus state: current_term: 1 leader_uuid: "ff89cefabae64b528906dfbdc4b8a692" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff89cefabae64b528906dfbdc4b8a692" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39485 } } }
I20260925 01:17:12.482781 25581 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:12.482340 25580 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff89cefabae64b528906dfbdc4b8a692" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff89cefabae64b528906dfbdc4b8a692" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39485 } } }
I20260925 01:17:12.483227 25580 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:12.483973 25587 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:17:12.489702 25587 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:17:12.498692 25587 catalog_manager.cc:1347] Generated new cluster ID: 77a9593be9104afab3b58c898592ce4e
I20260925 01:17:12.498821 25587 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:17:12.522394 25587 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:17:12.522862 25587 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:17:12.531240 25587 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692: Generated new TSK 0
I20260925 01:17:12.532217 25587 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:17:12.548779 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39485
--builtin_ntp_servers=127.20.102.148:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 01:17:12.894239 25598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:12.894527 25598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:12.894694 25598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:12.902199 25598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:12.902307 25598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:12.902460 25598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:17:12.911712 25598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:12.913097 25598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:12.915102 25598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:12.928637 25604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:12.928570 25606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:12.928573 25603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:12.929874 25598 server_base.cc:1034] running on GCE node
I20260925 01:17:12.930552 25598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:12.931433 25598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:12.932636 25598 hybrid_clock.cc:648] HybridClock initialized: now 1790299032932599 us; error 31 us; skew 500 ppm
I20260925 01:17:12.933000 25598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:17:12.935236 25598 webserver.cc:460] Webserver started at http://127.20.102.129:38597/ using document root <none> and password file <none>
I20260925 01:17:12.935925 25598 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:12.936043 25598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:12.936385 25598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:12.939224 25598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "9aad1ae1d0a141fb837e9ded24956c70"
format_stamp: "Formatted at 2026-09-25 01:17:12 on dist-test-slave-s12s"
server_key: "51622ace7b9f6125cb2b5680e3f81812"
server_key_iv: "c185ccbc85d181737e4911f546968748"
server_key_version: "encryptionkey@0"
I20260925 01:17:12.939996 25598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "9aad1ae1d0a141fb837e9ded24956c70"
format_stamp: "Formatted at 2026-09-25 01:17:12 on dist-test-slave-s12s"
server_key: "51622ace7b9f6125cb2b5680e3f81812"
server_key_iv: "c185ccbc85d181737e4911f546968748"
server_key_version: "encryptionkey@0"
I20260925 01:17:12.945710 25598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:17:12.950558 25612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:12.952087 25598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:17:12.952263 25598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "9aad1ae1d0a141fb837e9ded24956c70"
format_stamp: "Formatted at 2026-09-25 01:17:12 on dist-test-slave-s12s"
server_key: "51622ace7b9f6125cb2b5680e3f81812"
server_key_iv: "c185ccbc85d181737e4911f546968748"
server_key_version: "encryptionkey@0"
I20260925 01:17:12.952494 25598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:12.958153 25598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:12.958952 25598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:12.959235 25598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:12.960196 25598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:12.961658 25598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:12.961769 25598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:12.961900 25598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:12.962018 25598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:12.997496 25598 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:33921
I20260925 01:17:12.997545 25724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:33921 every 8 connection(s)
I20260925 01:17:12.998919 25598 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:17:13.001699 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 25598
I20260925 01:17:13.001905 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:17:13.002705 20890 external_mini_cluster.cc:1428] Setting key 7b4800e451b54b0fe1017caac9d23238
I20260925 01:17:13.005642 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39485
--builtin_ntp_servers=127.20.102.148:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 01:17:13.011786 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:13.001851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:43719 (local address 127.20.102.190:39485)
0925 01:17:13.009083 (+ 7232us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:13.009088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:13.009103 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:17:13.009170 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:13.009173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:13.009218 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:13.009305 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:13.010936 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:13.010946 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:13.010952 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:13.010975 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:13.011036 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:13.011054 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:13.011200 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:13.011388 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 01:17:13.011467 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7122,"thread_start_us":7004,"threads_started":1}
I20260925 01:17:13.012667 25728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:13.002617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39485 (local address 127.20.102.129:43719)
0925 01:17:13.004039 (+ 1422us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:13.004098 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:13.007145 (+ 3047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:13.009642 (+ 2497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:13.009658 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:13.009808 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:13.010571 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:17:13.010574 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:13.010616 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:17:13.010734 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:13.011175 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:13.011185 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:13.011190 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 01:17:13.011418 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 01:17:13.011643 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":386,"threads_started":1}
I20260925 01:17:13.014344 25725 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39485
I20260925 01:17:13.014680 25725 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:13.015386 25725 heartbeater.cc:511] Master 127.20.102.190:39485 requested a full tablet report, sending...
I20260925 01:17:13.017622 25539 ts_manager.cc:194] Registered new tserver with Master: 9aad1ae1d0a141fb837e9ded24956c70 (127.20.102.129:33921)
W20260925 01:17:13.330219 25729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:13.330502 25729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:13.330672 25729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:13.337852 25729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:13.337949 25729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:13.338153 25729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:17:13.346974 25729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:13.348248 25729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:13.350004 25729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:13.363102 25735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:13.363210 25737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:13.363189 25734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:13.364023 25729 server_base.cc:1034] running on GCE node
I20260925 01:17:13.364624 25729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:13.365574 25729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:13.366801 25729 hybrid_clock.cc:648] HybridClock initialized: now 1790299033366736 us; error 93 us; skew 500 ppm
I20260925 01:17:13.367118 25729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:17:13.369315 25729 webserver.cc:460] Webserver started at http://127.20.102.130:44445/ using document root <none> and password file <none>
I20260925 01:17:13.370002 25729 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:13.370163 25729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:13.370498 25729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:13.373288 25729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "8f4b4dfb76734a0da002b9c833c9a981"
format_stamp: "Formatted at 2026-09-25 01:17:13 on dist-test-slave-s12s"
server_key: "8a3e3243663fa31ba349e6cd18a4b3ce"
server_key_iv: "090d14465d085eb5ea272af000e7a326"
server_key_version: "encryptionkey@0"
I20260925 01:17:13.374059 25729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "8f4b4dfb76734a0da002b9c833c9a981"
format_stamp: "Formatted at 2026-09-25 01:17:13 on dist-test-slave-s12s"
server_key: "8a3e3243663fa31ba349e6cd18a4b3ce"
server_key_iv: "090d14465d085eb5ea272af000e7a326"
server_key_version: "encryptionkey@0"
I20260925 01:17:13.379240 25729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:17:13.382881 25743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:13.384145 25729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:17:13.384321 25729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "8f4b4dfb76734a0da002b9c833c9a981"
format_stamp: "Formatted at 2026-09-25 01:17:13 on dist-test-slave-s12s"
server_key: "8a3e3243663fa31ba349e6cd18a4b3ce"
server_key_iv: "090d14465d085eb5ea272af000e7a326"
server_key_version: "encryptionkey@0"
I20260925 01:17:13.384541 25729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:13.390336 25729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:13.391130 25729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:13.391397 25729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:13.392261 25729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:13.393672 25729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:13.393803 25729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:13.393926 25729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:13.394012 25729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:13.428175 25729 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:43229
I20260925 01:17:13.428264 25855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:43229 every 8 connection(s)
I20260925 01:17:13.429605 25729 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:17:13.437242 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 25729
I20260925 01:17:13.437454 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:17:13.437995 20890 external_mini_cluster.cc:1428] Setting key a01418694c1589318963cce7328e99e4
I20260925 01:17:13.438318 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:13.432440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:37721 (local address 127.20.102.190:39485)
0925 01:17:13.432618 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:13.432623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:13.434116 (+ 1493us) server_negotiation.cc:407] Connection header received
0925 01:17:13.435860 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:13.435865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:13.435956 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:13.436077 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:13.437539 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:13.437544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:13.437549 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:13.437567 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:13.437626 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:13.437641 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:13.437782 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:13.437971 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 01:17:13.438052 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:17:13.439446 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:13.433060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39485 (local address 127.20.102.130:37721)
0925 01:17:13.433952 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:13.433993 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:13.435585 (+ 1592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:13.436302 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:13.436310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:13.436398 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:13.437176 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:17:13.437178 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:13.437215 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:17:13.437298 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:13.437743 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:13.437754 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:13.437764 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 01:17:13.438055 (+ 291us) client_negotiation.cc:240] Negotiation successful
0925 01:17:13.438305 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":271,"threads_started":1}
I20260925 01:17:13.441022 25856 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39485
I20260925 01:17:13.441214 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39485
--builtin_ntp_servers=127.20.102.148:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 01:17:13.441310 25856 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:13.442003 25856 heartbeater.cc:511] Master 127.20.102.190:39485 requested a full tablet report, sending...
I20260925 01:17:13.443475 25539 ts_manager.cc:194] Registered new tserver with Master: 8f4b4dfb76734a0da002b9c833c9a981 (127.20.102.130:43229)
W20260925 01:17:13.765121 25859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:13.765439 25859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:13.765609 25859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:13.772775 25859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:13.772909 25859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:13.773111 25859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:17:13.782330 25859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:13.783702 25859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:13.785684 25859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:13.798691 25867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:13.798660 25864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:13.798772 25865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:13.799536 25859 server_base.cc:1034] running on GCE node
I20260925 01:17:13.800161 25859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:13.801155 25859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:13.802361 25859 hybrid_clock.cc:648] HybridClock initialized: now 1790299033802319 us; error 28 us; skew 500 ppm
I20260925 01:17:13.802704 25859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:17:13.805032 25859 webserver.cc:460] Webserver started at http://127.20.102.131:39795/ using document root <none> and password file <none>
I20260925 01:17:13.805714 25859 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:13.805830 25859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:13.806164 25859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:13.809017 25859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "b02d540449054c1da3bcc3365751eff8"
format_stamp: "Formatted at 2026-09-25 01:17:13 on dist-test-slave-s12s"
server_key: "3050db4b8a90b380f4537858997ebef8"
server_key_iv: "5df68f791f56534334a2ab0e515e6f0f"
server_key_version: "encryptionkey@0"
I20260925 01:17:13.809821 25859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "b02d540449054c1da3bcc3365751eff8"
format_stamp: "Formatted at 2026-09-25 01:17:13 on dist-test-slave-s12s"
server_key: "3050db4b8a90b380f4537858997ebef8"
server_key_iv: "5df68f791f56534334a2ab0e515e6f0f"
server_key_version: "encryptionkey@0"
I20260925 01:17:13.814935 25859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:17:13.818435 25873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:13.819794 25859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:17:13.819967 25859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "b02d540449054c1da3bcc3365751eff8"
format_stamp: "Formatted at 2026-09-25 01:17:13 on dist-test-slave-s12s"
server_key: "3050db4b8a90b380f4537858997ebef8"
server_key_iv: "5df68f791f56534334a2ab0e515e6f0f"
server_key_version: "encryptionkey@0"
I20260925 01:17:13.820158 25859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:13.825706 25859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:13.826404 25859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:13.826633 25859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:13.827435 25859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:13.828789 25859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:13.828876 25859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:13.828956 25859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:13.829011 25859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:13.861075 25859 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:35329
I20260925 01:17:13.861145 25986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:35329 every 8 connection(s)
I20260925 01:17:13.862387 25859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:17:13.870759 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:13.865022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:50929 (local address 127.20.102.190:39485)
0925 01:17:13.865196 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:13.865201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:13.866564 (+ 1363us) server_negotiation.cc:407] Connection header received
0925 01:17:13.868540 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:13.868546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:13.868631 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:13.868777 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:13.869988 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:13.869994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:13.869998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:13.870026 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:13.870071 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:13.870083 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:13.870221 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:13.870416 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 01:17:13.870500 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:17:13.871742 25989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:13.865496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39485 (local address 127.20.102.131:50929)
0925 01:17:13.866382 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:13.866417 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:13.868241 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:13.868995 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:13.869003 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:13.869092 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:13.869693 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:17:13.869695 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:13.869731 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:17:13.869813 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:13.870180 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:13.870190 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:13.870200 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 01:17:13.870508 (+ 308us) client_negotiation.cc:240] Negotiation successful
0925 01:17:13.870751 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":257,"threads_started":1}
I20260925 01:17:13.872018 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 25859
I20260925 01:17:13.872195 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:17:13.872670 20890 external_mini_cluster.cc:1428] Setting key 1a7af161a0ba99aade795272b35494d2
I20260925 01:17:13.873246 25987 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39485
I20260925 01:17:13.873557 25987 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:13.874231 25987 heartbeater.cc:511] Master 127.20.102.190:39485 requested a full tablet report, sending...
I20260925 01:17:13.875676 25539 ts_manager.cc:194] Registered new tserver with Master: b02d540449054c1da3bcc3365751eff8 (127.20.102.131:35329)
I20260925 01:17:13.887959 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:17:13.895395 25727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:13.892761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40654 (local address 127.20.102.190:39485)
0925 01:17:13.892986 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:13.892995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:13.893465 (+ 470us) server_negotiation.cc:407] Connection header received
0925 01:17:13.893610 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:13.893614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:13.893666 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:13.893747 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:13.894796 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:13.894800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:13.894803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:13.894816 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:13.894842 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:13.894853 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:13.894940 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:13.895031 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 01:17:13.895083 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:17:13.899359 25539 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:17:13.906119 25539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:17:13.933871 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:13.921954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:35329 (local address 127.0.0.1:36476)
0925 01:17:13.930919 (+ 8965us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:13.930936 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:13.931063 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:13.931830 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:13.931834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:13.931855 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:13.932756 (+ 901us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:17:13.932759 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:13.932773 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:17:13.932815 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:13.933178 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:13.933188 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:13.933196 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 01:17:13.933361 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 01:17:13.933447 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8836,"thread_start_us":7016,"threads_started":1}
I20260925 01:17:13.933871 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:13.922543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36476 (local address 127.20.102.131:35329)
0925 01:17:13.923268 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:13.923280 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:13.931045 (+ 7765us) server_negotiation.cc:407] Connection header received
0925 01:17:13.931315 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:13.931325 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:13.931518 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:13.931690 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:13.932971 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:13.932979 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:13.932991 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:13.933014 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:13.933061 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:13.933075 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:13.933260 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:13.933337 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 01:17:13.933440 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":381,"threads_started":1}
I20260925 01:17:13.933934 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:13.919584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.20.102.129:33921)
0925 01:17:13.920209 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:13.920223 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:13.930429 (+ 10206us) server_negotiation.cc:407] Connection header received
0925 01:17:13.930667 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:13.930677 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:13.930912 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:13.931052 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:13.932654 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:13.932667 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:13.932710 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:13.933016 (+ 306us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:13.933086 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:13.933109 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:13.933247 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:13.933469 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 01:17:13.933571 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":285,"threads_started":1}
I20260925 01:17:13.934401 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:13.919068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.20.102.130:43229)
0925 01:17:13.930193 (+ 11125us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:13.930201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:13.930231 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:17:13.930311 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:13.930322 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:13.930618 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:13.930763 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:13.933279 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:13.933287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:13.933298 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:13.933325 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:13.933382 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:13.933406 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:13.933555 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:13.933802 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 01:17:13.933885 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10946,"thread_start_us":11188,"threads_started":1}
I20260925 01:17:13.934701 26003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:13.918901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:33921 (local address 127.0.0.1:37262)
0925 01:17:13.930299 (+ 11398us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:13.930323 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:13.930508 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:13.931270 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:13.931277 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:13.931379 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:13.932343 (+ 964us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:17:13.932347 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:13.932382 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:17:13.932476 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:13.933297 (+ 821us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:13.933301 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:13.933308 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 01:17:13.934216 (+ 908us) client_negotiation.cc:240] Negotiation successful
0925 01:17:13.934277 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10220,"thread_start_us":10730,"threads_started":1}
I20260925 01:17:13.935603 25999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:13.918929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:43229 (local address 127.0.0.1:53798)
0925 01:17:13.929922 (+ 10993us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:13.929981 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:13.930150 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:13.932315 (+ 2165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:13.932322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:13.932349 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:13.933059 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:17:13.933062 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:13.933077 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:17:13.933117 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:13.933645 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:13.933650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:13.933654 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 01:17:13.935074 (+ 1420us) client_negotiation.cc:240] Negotiation successful
0925 01:17:13.935123 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10760,"thread_start_us":11729,"threads_started":1}
I20260925 01:17:13.938720 25660 tablet_service.cc:1467] Processing CreateTablet for tablet d0468b491f174ec6a7b601a5f2c926dd (DEFAULT_TABLE table=test-table [id=487ed8cf412d4c248fb43ea8c64b860b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:13.938720 25921 tablet_service.cc:1467] Processing CreateTablet for tablet d0468b491f174ec6a7b601a5f2c926dd (DEFAULT_TABLE table=test-table [id=487ed8cf412d4c248fb43ea8c64b860b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:13.939055 25791 tablet_service.cc:1467] Processing CreateTablet for tablet d0468b491f174ec6a7b601a5f2c926dd (DEFAULT_TABLE table=test-table [id=487ed8cf412d4c248fb43ea8c64b860b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:13.940187 25791 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0468b491f174ec6a7b601a5f2c926dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:13.940187 25660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0468b491f174ec6a7b601a5f2c926dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:13.940196 25921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0468b491f174ec6a7b601a5f2c926dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:13.953441 26005 tablet_bootstrap.cc:492] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70: Bootstrap starting.
I20260925 01:17:13.957103 26005 tablet_bootstrap.cc:654] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:13.957742 26007 tablet_bootstrap.cc:492] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8: Bootstrap starting.
I20260925 01:17:13.958254 26005 log.cc:824] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:13.961519 26006 tablet_bootstrap.cc:492] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981: Bootstrap starting.
I20260925 01:17:13.963027 26007 tablet_bootstrap.cc:654] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:13.964046 26007 log.cc:824] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:13.966130 26006 tablet_bootstrap.cc:654] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:13.967043 26006 log.cc:824] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:13.967254 26005 tablet_bootstrap.cc:492] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70: No bootstrap required, opened a new log
I20260925 01:17:13.967254 26007 tablet_bootstrap.cc:492] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8: No bootstrap required, opened a new log
I20260925 01:17:13.967528 26007 ts_tablet_manager.cc:1397] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 01:17:13.967528 26005 ts_tablet_manager.cc:1397] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260925 01:17:13.969596 26006 tablet_bootstrap.cc:492] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981: No bootstrap required, opened a new log
I20260925 01:17:13.969847 26006 ts_tablet_manager.cc:1397] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 01:17:13.977617 26006 raft_consensus.cc:348] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4b4dfb76734a0da002b9c833c9a981" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43229 } } peers { permanent_uuid: "9aad1ae1d0a141fb837e9ded24956c70" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 33921 } } peers { permanent_uuid: "b02d540449054c1da3bcc3365751eff8" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 35329 } }
I20260925 01:17:13.977998 26006 raft_consensus.cc:374] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:13.978128 26006 raft_consensus.cc:729] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f4b4dfb76734a0da002b9c833c9a981, State: Initialized, Role: FOLLOWER
I20260925 01:17:13.978297 26005 raft_consensus.cc:348] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4b4dfb76734a0da002b9c833c9a981" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43229 } } peers { permanent_uuid: "9aad1ae1d0a141fb837e9ded24956c70" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 33921 } } peers { permanent_uuid: "b02d540449054c1da3bcc3365751eff8" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 35329 } }
I20260925 01:17:13.978297 26007 raft_consensus.cc:348] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4b4dfb76734a0da002b9c833c9a981" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43229 } } peers { permanent_uuid: "9aad1ae1d0a141fb837e9ded24956c70" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 33921 } } peers { permanent_uuid: "b02d540449054c1da3bcc3365751eff8" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 35329 } }
I20260925 01:17:13.978713 26005 raft_consensus.cc:374] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:13.978713 26007 raft_consensus.cc:374] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:13.978823 26007 raft_consensus.cc:729] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b02d540449054c1da3bcc3365751eff8, State: Initialized, Role: FOLLOWER
I20260925 01:17:13.978817 26005 raft_consensus.cc:729] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aad1ae1d0a141fb837e9ded24956c70, State: Initialized, Role: FOLLOWER
I20260925 01:17:13.978777 26006 consensus_queue.cc:260] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4b4dfb76734a0da002b9c833c9a981" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43229 } } peers { permanent_uuid: "9aad1ae1d0a141fb837e9ded24956c70" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 33921 } } peers { permanent_uuid: "b02d540449054c1da3bcc3365751eff8" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 35329 } }
I20260925 01:17:13.979480 26005 consensus_queue.cc:260] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4b4dfb76734a0da002b9c833c9a981" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43229 } } peers { permanent_uuid: "9aad1ae1d0a141fb837e9ded24956c70" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 33921 } } peers { permanent_uuid: "b02d540449054c1da3bcc3365751eff8" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 35329 } }
I20260925 01:17:13.979460 26007 consensus_queue.cc:260] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4b4dfb76734a0da002b9c833c9a981" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43229 } } peers { permanent_uuid: "9aad1ae1d0a141fb837e9ded24956c70" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 33921 } } peers { permanent_uuid: "b02d540449054c1da3bcc3365751eff8" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 35329 } }
I20260925 01:17:13.980095 25856 heartbeater.cc:503] Master 127.20.102.190:39485 was elected leader, sending a full tablet report...
I20260925 01:17:13.980639 26006 ts_tablet_manager.cc:1428] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260925 01:17:13.981721 25725 heartbeater.cc:503] Master 127.20.102.190:39485 was elected leader, sending a full tablet report...
I20260925 01:17:13.982043 25987 heartbeater.cc:503] Master 127.20.102.190:39485 was elected leader, sending a full tablet report...
I20260925 01:17:13.982272 26005 ts_tablet_manager.cc:1428] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
I20260925 01:17:13.982609 26007 ts_tablet_manager.cc:1428] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260925 01:17:14.001238 25726 tablet.cc:2369] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:17:14.106033 26012 raft_consensus.cc:482] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:17:14.106431 26012 raft_consensus.cc:504] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4b4dfb76734a0da002b9c833c9a981" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43229 } } peers { permanent_uuid: "9aad1ae1d0a141fb837e9ded24956c70" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 33921 } } peers { permanent_uuid: "b02d540449054c1da3bcc3365751eff8" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 35329 } }
I20260925 01:17:14.108397 26012 leader_election.cc:288] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f4b4dfb76734a0da002b9c833c9a981 (127.20.102.130:43229), b02d540449054c1da3bcc3365751eff8 (127.20.102.131:35329)
I20260925 01:17:14.112015 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.108899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:46107 (local address 127.20.102.130:43229)
0925 01:17:14.109131 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:14.109139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:14.109891 (+ 752us) server_negotiation.cc:407] Connection header received
0925 01:17:14.110046 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:14.110055 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:14.110131 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:14.110237 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:14.111309 (+ 1072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:14.111315 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:14.111320 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:14.111339 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:14.111380 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:14.111400 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:14.111542 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:14.111605 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 01:17:14.111683 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 01:17:14.112164 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.109190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:53573 (local address 127.20.102.131:35329)
0925 01:17:14.109415 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:14.109423 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:14.110125 (+ 702us) server_negotiation.cc:407] Connection header received
0925 01:17:14.110255 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:14.110260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:14.110320 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:14.110430 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:14.111517 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:14.111522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:14.111527 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:14.111548 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:14.111588 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:14.111604 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:14.111717 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:14.111860 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 01:17:14.111918 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 01:17:14.112183 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.108837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:43229 (local address 127.20.102.129:46107)
0925 01:17:14.109725 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:14.109770 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:14.109911 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:14.110268 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:14.110272 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:14.110296 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:14.111069 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:17:14.111073 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:14.111094 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:17:14.111144 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:14.111502 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:14.111506 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:14.111510 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 01:17:14.111643 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 01:17:14.111697 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":487,"threads_started":1}
I20260925 01:17:14.113034 26015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.108783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:35329 (local address 127.20.102.129:53573)
0925 01:17:14.109999 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:14.110017 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:14.110131 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:14.110469 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:14.110473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:14.110495 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:14.111273 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:17:14.111277 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:14.111293 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:17:14.111336 (+ 43us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:14.111713 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:14.111718 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:14.111723 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 01:17:14.111906 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 01:17:14.111957 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":555,"threads_started":1}
I20260925 01:17:14.113210 25811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0468b491f174ec6a7b601a5f2c926dd" candidate_uuid: "9aad1ae1d0a141fb837e9ded24956c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4b4dfb76734a0da002b9c833c9a981" is_pre_election: true
I20260925 01:17:14.113761 25811 raft_consensus.cc:2454] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aad1ae1d0a141fb837e9ded24956c70 in term 0.
W20260925 01:17:14.114079 25988 tablet.cc:2369] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:17:14.114197 25942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0468b491f174ec6a7b601a5f2c926dd" candidate_uuid: "9aad1ae1d0a141fb837e9ded24956c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b02d540449054c1da3bcc3365751eff8" is_pre_election: true
I20260925 01:17:14.114441 25613 leader_election.cc:302] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f4b4dfb76734a0da002b9c833c9a981, 9aad1ae1d0a141fb837e9ded24956c70; no voters:
I20260925 01:17:14.114861 26012 raft_consensus.cc:2781] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:17:14.114789 25942 raft_consensus.cc:2454] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aad1ae1d0a141fb837e9ded24956c70 in term 0.
I20260925 01:17:14.115017 26012 raft_consensus.cc:482] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:17:14.115118 26012 raft_consensus.cc:3037] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:14.117905 26012 raft_consensus.cc:504] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4b4dfb76734a0da002b9c833c9a981" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43229 } } peers { permanent_uuid: "9aad1ae1d0a141fb837e9ded24956c70" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 33921 } } peers { permanent_uuid: "b02d540449054c1da3bcc3365751eff8" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 35329 } }
I20260925 01:17:14.118562 26012 leader_election.cc:288] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [CANDIDATE]: Term 1 election: Requested vote from peers 8f4b4dfb76734a0da002b9c833c9a981 (127.20.102.130:43229), b02d540449054c1da3bcc3365751eff8 (127.20.102.131:35329)
I20260925 01:17:14.118960 25811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0468b491f174ec6a7b601a5f2c926dd" candidate_uuid: "9aad1ae1d0a141fb837e9ded24956c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f4b4dfb76734a0da002b9c833c9a981"
I20260925 01:17:14.119166 25811 raft_consensus.cc:3037] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:14.119143 25942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0468b491f174ec6a7b601a5f2c926dd" candidate_uuid: "9aad1ae1d0a141fb837e9ded24956c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b02d540449054c1da3bcc3365751eff8"
I20260925 01:17:14.119362 25942 raft_consensus.cc:3037] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:14.121711 25811 raft_consensus.cc:2454] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aad1ae1d0a141fb837e9ded24956c70 in term 1.
I20260925 01:17:14.122088 25942 raft_consensus.cc:2454] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aad1ae1d0a141fb837e9ded24956c70 in term 1.
I20260925 01:17:14.122119 25613 leader_election.cc:302] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f4b4dfb76734a0da002b9c833c9a981, 9aad1ae1d0a141fb837e9ded24956c70; no voters:
I20260925 01:17:14.122452 26012 raft_consensus.cc:2781] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:14.122921 26012 raft_consensus.cc:686] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [term 1 LEADER]: Becoming Leader. State: Replica: 9aad1ae1d0a141fb837e9ded24956c70, State: Running, Role: LEADER
I20260925 01:17:14.123530 26012 consensus_queue.cc:237] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4b4dfb76734a0da002b9c833c9a981" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43229 } } peers { permanent_uuid: "9aad1ae1d0a141fb837e9ded24956c70" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 33921 } } peers { permanent_uuid: "b02d540449054c1da3bcc3365751eff8" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 35329 } }
I20260925 01:17:14.128749 25538 catalog_manager.cc:5495] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9aad1ae1d0a141fb837e9ded24956c70 (127.20.102.129). New cstate: current_term: 1 leader_uuid: "9aad1ae1d0a141fb837e9ded24956c70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f4b4dfb76734a0da002b9c833c9a981" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9aad1ae1d0a141fb837e9ded24956c70" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 33921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b02d540449054c1da3bcc3365751eff8" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 35329 } health_report { overall_health: UNKNOWN } } }
I20260925 01:17:14.172464 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.170369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37264 (local address 127.20.102.129:33921)
0925 01:17:14.170544 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:14.170549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:14.170723 (+ 174us) server_negotiation.cc:407] Connection header received
0925 01:17:14.170910 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:14.170913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:14.170994 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:14.171080 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:14.171960 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:14.171965 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:14.171968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:14.171983 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:14.172014 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:14.172025 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:14.172118 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:14.172208 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 01:17:14.172253 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:17:14.180832 25942 raft_consensus.cc:1261] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8 [term 1 FOLLOWER]: Refusing update from remote peer 9aad1ae1d0a141fb837e9ded24956c70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:14.180876 25811 raft_consensus.cc:1261] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981 [term 1 FOLLOWER]: Refusing update from remote peer 9aad1ae1d0a141fb837e9ded24956c70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 01:17:14.181597 25857 tablet.cc:2369] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:17:14.181779 26016 consensus_queue.cc:1035] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b02d540449054c1da3bcc3365751eff8" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 35329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:14.182197 26012 consensus_queue.cc:1035] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f4b4dfb76734a0da002b9c833c9a981" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:14.195569 26019 mvcc.cc:205] Tried to move back new op lower bound from 7333064843994669056 to 7333064843774271488. Current Snapshot: MvccSnapshot[applied={T|T < 7333064843994669056}]
I20260925 01:17:14.209306 25538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:14.209636 25538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40654:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:14.213325 25538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ff89cefabae64b528906dfbdc4b8a692: Sending DeleteTablet for 3 replicas of tablet d0468b491f174ec6a7b601a5f2c926dd
I20260925 01:17:14.214495 25660 tablet_service.cc:1514] Processing DeleteTablet for tablet d0468b491f174ec6a7b601a5f2c926dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:14 UTC) from {username='slave'} at 127.0.0.1:37262
I20260925 01:17:14.214856 25791 tablet_service.cc:1514] Processing DeleteTablet for tablet d0468b491f174ec6a7b601a5f2c926dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:14 UTC) from {username='slave'} at 127.0.0.1:53798
I20260925 01:17:14.215143 25921 tablet_service.cc:1514] Processing DeleteTablet for tablet d0468b491f174ec6a7b601a5f2c926dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:14 UTC) from {username='slave'} at 127.0.0.1:36476
I20260925 01:17:14.216740 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 25598
I20260925 01:17:14.225208 26028 tablet_replica.cc:330] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70: stopping tablet replica
I20260925 01:17:14.225611 26028 raft_consensus.cc:2229] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:17:14.226238 26028 raft_consensus.cc:2258] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:14.227420 26002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.220823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37272 (local address 127.20.102.129:33921)
0925 01:17:14.221070 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:14.221080 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:14.223843 (+ 2763us) server_negotiation.cc:407] Connection header received
0925 01:17:14.224760 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:14.224770 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:14.224853 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:14.224951 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:14.226544 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:14.226552 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:14.226562 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:14.226585 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:14.226625 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:14.226640 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:14.226773 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:14.226998 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 01:17:14.227090 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 01:17:14.228888 25720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:14.232800 26029 tablet_replica.cc:330] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981: stopping tablet replica
I20260925 01:17:14.233266 26029 raft_consensus.cc:2229] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:14.233806 26029 raft_consensus.cc:2258] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:14.233908 26030 tablet_replica.cc:330] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8: stopping tablet replica
I20260925 01:17:14.234315 26030 raft_consensus.cc:2229] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:14.234789 26030 raft_consensus.cc:2258] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:14.236547 26029 ts_tablet_manager.cc:1905] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:14.236714 26030 ts_tablet_manager.cc:1905] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:14.243804 26029 ts_tablet_manager.cc:1918] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:14.243804 26030 ts_tablet_manager.cc:1918] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:14.243966 26029 log.cc:1196] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/d0468b491f174ec6a7b601a5f2c926dd
I20260925 01:17:14.243973 26030 log.cc:1196] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/d0468b491f174ec6a7b601a5f2c926dd
I20260925 01:17:14.244506 26030 ts_tablet_manager.cc:1939] T d0468b491f174ec6a7b601a5f2c926dd P b02d540449054c1da3bcc3365751eff8: Deleting consensus metadata
I20260925 01:17:14.244514 26029 ts_tablet_manager.cc:1939] T d0468b491f174ec6a7b601a5f2c926dd P 8f4b4dfb76734a0da002b9c833c9a981: Deleting consensus metadata
I20260925 01:17:14.246469 25524 catalog_manager.cc:4841] TS b02d540449054c1da3bcc3365751eff8 (127.20.102.131:35329): tablet d0468b491f174ec6a7b601a5f2c926dd (table test-table [id=487ed8cf412d4c248fb43ea8c64b860b]) successfully deleted
I20260925 01:17:14.246416 25523 catalog_manager.cc:4841] TS 8f4b4dfb76734a0da002b9c833c9a981 (127.20.102.130:43229): tablet d0468b491f174ec6a7b601a5f2c926dd (table test-table [id=487ed8cf412d4c248fb43ea8c64b860b]) successfully deleted
I20260925 01:17:14.229250 26028 ts_tablet_manager.cc:1905] T d0468b491f174ec6a7b601a5f2c926dd P 9aad1ae1d0a141fb837e9ded24956c70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:14.325779 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 25598
I20260925 01:17:14.344184 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 25729
W20260925 01:17:14.344285 25525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.102.129:33921 (error 108)
I20260925 01:17:14.346498 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.345959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:33921 (local address 127.0.0.1:37286)
0925 01:17:14.346155 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:14.346302 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 01:17:14.346758 25525 catalog_manager.cc:4584] TS 9aad1ae1d0a141fb837e9ded24956c70 (127.20.102.129:33921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d0468b491f174ec6a7b601a5f2c926dd: Network error: Client connection negotiation failed: client connection to 127.20.102.129:33921: connect: Connection refused (error 111)
I20260925 01:17:14.347455 26001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.345216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53802 (local address 127.20.102.130:43229)
0925 01:17:14.345412 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:14.345420 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:14.345444 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:17:14.345554 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:14.345560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:14.345676 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:14.345798 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:14.346766 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:14.346774 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:14.346782 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:14.346802 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:14.346839 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:14.346855 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:14.347025 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:14.347080 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 01:17:14.347137 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:17:14.348129 25851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:14.400316 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.399835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:33921 (local address 127.0.0.1:37294)
0925 01:17:14.400052 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:14.400149 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 01:17:14.401391 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.401035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:33921 (local address 127.0.0.1:37306)
0925 01:17:14.401196 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:14.401277 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 01:17:14.437302 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 25729
I20260925 01:17:14.454908 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 25859
I20260925 01:17:14.458055 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.455768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.20.102.131:35329)
0925 01:17:14.456007 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:14.456014 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:14.456035 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:17:14.456221 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:14.456226 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:14.456307 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:14.456429 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:14.457410 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:14.457416 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:14.457421 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:14.457445 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:14.457476 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:14.457489 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:14.457623 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:14.457711 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 01:17:14.457772 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:17:14.458690 25982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:14.462558 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.462064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:33921 (local address 127.0.0.1:37322)
0925 01:17:14.462244 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:14.462363 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 01:17:14.463528 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.463191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:33921 (local address 127.0.0.1:37334)
0925 01:17:14.463341 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:14.463418 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 01:17:14.544123 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.543626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:33921 (local address 127.0.0.1:37336)
0925 01:17:14.543857 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:14.543953 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 01:17:14.545398 26000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:14.544900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:33921 (local address 127.0.0.1:37348)
0925 01:17:14.545107 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:14.545234 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:33921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 01:17:14.547154 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 25859
I20260925 01:17:14.564623 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 25508
I20260925 01:17:14.566006 25569 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:14.672263 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 25508
2026-09-25T01:17:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2713 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 01:17:14.714040 20890 mini_oidc.cc:152] Starting JWKS server
I20260925 01:17:14.716522 20890 webserver.cc:460] Webserver started at https://127.0.0.1:36637/ using document root <none> and password file <none>
I20260925 01:17:14.716846 20890 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 01:17:14.717427 20890 webserver.cc:460] Webserver started at http://127.0.0.1:42609/ using document root <none> and password file <none>
2026-09-25T01:17:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:17:14Z Disabled control of system clock
I20260925 01:17:14.745371 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:45187
--webserver_interface=127.20.102.190
--webserver_port=0
--builtin_ntp_servers=127.20.102.148:45593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:45187
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36637/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/testchainca.cert with env {}
W20260925 01:17:15.088866 26042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:15.089151 26042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:15.089238 26042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:15.096534 26042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:15.096645 26042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:15.096746 26042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:15.096812 26042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:15.096858 26042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 01:17:15.096920 26042 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 01:17:15.096958 26042 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36637/jwks.json
I20260925 01:17:15.106751 26042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:45593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:45187
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:45187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36637/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:15.108328 26042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:15.109854 26042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:15.121999 26048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:15.121963 26050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:15.122048 26047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:15.122933 26042 server_base.cc:1034] running on GCE node
I20260925 01:17:15.123672 26042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:15.125099 26042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:15.126300 26042 hybrid_clock.cc:648] HybridClock initialized: now 1790299035126251 us; error 43 us; skew 500 ppm
I20260925 01:17:15.126668 26042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:17:15.128871 26042 webserver.cc:460] Webserver started at http://127.20.102.190:37213/ using document root <none> and password file <none>
I20260925 01:17:15.129539 26042 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:15.129659 26042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:15.129984 26042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:15.132999 26042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "e05a7caecb2e46f9b6bfcab538b30675"
format_stamp: "Formatted at 2026-09-25 01:17:15 on dist-test-slave-s12s"
server_key: "6982054e2525c458a3c648d66e7d8913"
server_key_iv: "a0e7c320356a01e54b519c706533874d"
server_key_version: "encryptionkey@0"
I20260925 01:17:15.133862 26042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "e05a7caecb2e46f9b6bfcab538b30675"
format_stamp: "Formatted at 2026-09-25 01:17:15 on dist-test-slave-s12s"
server_key: "6982054e2525c458a3c648d66e7d8913"
server_key_iv: "a0e7c320356a01e54b519c706533874d"
server_key_version: "encryptionkey@0"
I20260925 01:17:15.138896 26042 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 01:17:15.142230 26056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:15.143507 26042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:17:15.143697 26042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "e05a7caecb2e46f9b6bfcab538b30675"
format_stamp: "Formatted at 2026-09-25 01:17:15 on dist-test-slave-s12s"
server_key: "6982054e2525c458a3c648d66e7d8913"
server_key_iv: "a0e7c320356a01e54b519c706533874d"
server_key_version: "encryptionkey@0"
I20260925 01:17:15.143905 26042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:15.157541 26042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:15.158270 26042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:15.158564 26042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:15.176394 26042 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:45187
I20260925 01:17:15.176416 26107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:45187 every 8 connection(s)
I20260925 01:17:15.177767 26042 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:17:15.181641 26108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:15.182387 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 26042
I20260925 01:17:15.182621 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:17:15.183416 20890 external_mini_cluster.cc:1428] Setting key 43a82f640f0fee7289ec62fc4457a339
I20260925 01:17:15.194402 26108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675: Bootstrap starting.
I20260925 01:17:15.196617 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:15.185335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54182 (local address 127.20.102.190:45187)
0925 01:17:15.186281 (+ 946us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:15.186302 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:15.186340 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:17:15.188098 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:15.188119 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:15.188582 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:15.189027 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:15.189602 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:15.190874 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:15.192262 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:15.192994 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:15.193194 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:15.194405 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:15.194413 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:15.194427 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:15.194986 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:15.195061 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:15.195081 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:15.195223 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:15.195439 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 01:17:15.195732 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":327,"threads_started":1}
I20260925 01:17:15.197664 26108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:15.198691 26108 log.cc:824] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:15.200448 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:17:15.201895 26108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675: No bootstrap required, opened a new log
I20260925 01:17:15.209118 26108 raft_consensus.cc:348] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05a7caecb2e46f9b6bfcab538b30675" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45187 } }
I20260925 01:17:15.209498 26108 raft_consensus.cc:374] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:15.209621 26108 raft_consensus.cc:729] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e05a7caecb2e46f9b6bfcab538b30675, State: Initialized, Role: FOLLOWER
I20260925 01:17:15.210134 26108 consensus_queue.cc:260] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05a7caecb2e46f9b6bfcab538b30675" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45187 } }
I20260925 01:17:15.210366 26108 raft_consensus.cc:388] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:17:15.210512 26108 raft_consensus.cc:482] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:17:15.210696 26108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:15.212773 26108 raft_consensus.cc:504] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05a7caecb2e46f9b6bfcab538b30675" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45187 } }
I20260925 01:17:15.213258 26108 leader_election.cc:302] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e05a7caecb2e46f9b6bfcab538b30675; no voters:
I20260925 01:17:15.213832 26108 leader_election.cc:288] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:17:15.213982 26113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:15.214723 26113 raft_consensus.cc:686] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [term 1 LEADER]: Becoming Leader. State: Replica: e05a7caecb2e46f9b6bfcab538b30675, State: Running, Role: LEADER
I20260925 01:17:15.215245 26113 consensus_queue.cc:237] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05a7caecb2e46f9b6bfcab538b30675" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45187 } }
I20260925 01:17:15.215947 26108 sys_catalog.cc:564] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:15.218154 26115 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e05a7caecb2e46f9b6bfcab538b30675. Latest consensus state: current_term: 1 leader_uuid: "e05a7caecb2e46f9b6bfcab538b30675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05a7caecb2e46f9b6bfcab538b30675" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45187 } } }
I20260925 01:17:15.218128 26114 sys_catalog.cc:455] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e05a7caecb2e46f9b6bfcab538b30675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e05a7caecb2e46f9b6bfcab538b30675" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45187 } } }
I20260925 01:17:15.218600 26115 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:15.218600 26114 sys_catalog.cc:458] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:15.221696 26118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:17:15.227649 26118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:17:15.237298 26118 catalog_manager.cc:1347] Generated new cluster ID: 18552795888a4e35ae8eb427ac2802a8
I20260925 01:17:15.237423 26118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:17:15.251682 26118 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:17:15.252954 26118 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:17:15.259733 26118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e05a7caecb2e46f9b6bfcab538b30675: Generated new TSK 0
I20260925 01:17:15.260757 26118 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 01:17:15.303768 26138 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 01:17:15.303936 26138 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 01:17:15.305251 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:15.283397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.20.102.190:45187)
0925 01:17:15.283603 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:15.283611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:15.284018 (+ 407us) server_negotiation.cc:407] Connection header received
0925 01:17:15.284288 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:15.284296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:15.284407 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:15.284600 (+ 193us) server_negotiation.cc:226] Negotiated authn=JWT
0925 01:17:15.285159 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:15.286330 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:15.287687 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:15.288476 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:15.288636 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:15.289205 (+ 569us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 01:17:15.304466 (+ 15261us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 01:17:15.304904 (+ 438us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54190: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":83}
W20260925 01:17:15.305378 26111 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54190: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 01:17:15.306560 26137 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:15.283265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:45187 (local address 127.0.0.1:54190)
0925 01:17:15.283909 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:15.283929 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:15.284068 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:15.284616 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:15.284619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:15.284629 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0925 01:17:15.285001 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:15.285010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:15.286473 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:15.286476 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:15.287519 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:15.287530 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:15.288640 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:15.288644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:15.288849 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:15.288916 (+ 67us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 01:17:15.304793 (+ 15877us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 01:17:15.305042 (+ 249us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.102.190:45187: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":337,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.20.102.190:45187: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36637/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 01:17:15.308377 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 26042
I20260925 01:17:15.309504 26103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:15.403120 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 26042
2026-09-25T01:17:15Z chronyd exiting
I20260925 01:17:15.454336 20890 test_util.cc:175] -----------------------------------------------
I20260925 01:17:15.454592 20890 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790298989998230-20890-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (744 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 01:17:15.458529 20890 mini_oidc.cc:152] Starting JWKS server
I20260925 01:17:15.460421 20890 webserver.cc:460] Webserver started at https://127.0.0.1:41875/ using document root <none> and password file <none>
I20260925 01:17:15.460708 20890 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 01:17:15.461333 20890 webserver.cc:460] Webserver started at http://127.0.0.1:46455/ using document root <none> and password file <none>
2026-09-25T01:17:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:17:15Z Disabled control of system clock
I20260925 01:17:15.487948 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:36745
--webserver_interface=127.20.102.190
--webserver_port=0
--builtin_ntp_servers=127.20.102.148:38267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:36745
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41875/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/testchainca.cert with env {}
W20260925 01:17:15.811379 26145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:15.811703 26145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:15.811865 26145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:15.818954 26145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:15.819087 26145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:15.819172 26145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:15.819283 26145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:15.819360 26145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 01:17:15.819453 26145 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 01:17:15.819530 26145 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41875/jwks.json
I20260925 01:17:15.829077 26145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:36745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:36745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41875/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:15.830370 26145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:15.831687 26145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:15.842955 26150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:15.843016 26153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:15.843052 26151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:15.844452 26145 server_base.cc:1034] running on GCE node
I20260925 01:17:15.845292 26145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:15.846629 26145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:15.847863 26145 hybrid_clock.cc:648] HybridClock initialized: now 1790299035847827 us; error 50 us; skew 500 ppm
I20260925 01:17:15.848202 26145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:17:15.850596 26145 webserver.cc:460] Webserver started at http://127.20.102.190:40377/ using document root <none> and password file <none>
I20260925 01:17:15.851264 26145 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:15.851398 26145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:15.851722 26145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:15.854509 26145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "11f29479fd97431fad2b35af0bdb328d"
format_stamp: "Formatted at 2026-09-25 01:17:15 on dist-test-slave-s12s"
server_key: "73dfa0c3a4c98548290291af91f17e28"
server_key_iv: "fdd51a6294fcda4cf395596175369e9f"
server_key_version: "encryptionkey@0"
I20260925 01:17:15.855290 26145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "11f29479fd97431fad2b35af0bdb328d"
format_stamp: "Formatted at 2026-09-25 01:17:15 on dist-test-slave-s12s"
server_key: "73dfa0c3a4c98548290291af91f17e28"
server_key_iv: "fdd51a6294fcda4cf395596175369e9f"
server_key_version: "encryptionkey@0"
I20260925 01:17:15.860110 26145 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260925 01:17:15.863513 26159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:15.864840 26145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:15.865027 26145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "11f29479fd97431fad2b35af0bdb328d"
format_stamp: "Formatted at 2026-09-25 01:17:15 on dist-test-slave-s12s"
server_key: "73dfa0c3a4c98548290291af91f17e28"
server_key_iv: "fdd51a6294fcda4cf395596175369e9f"
server_key_version: "encryptionkey@0"
I20260925 01:17:15.865275 26145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:15.878681 26145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:15.879407 26145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:15.879706 26145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:15.896726 26145 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:36745
I20260925 01:17:15.896723 26210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:36745 every 8 connection(s)
I20260925 01:17:15.898013 26145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:17:15.901648 26211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:15.905293 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 26145
I20260925 01:17:15.905516 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:17:15.906162 20890 external_mini_cluster.cc:1428] Setting key 59f58ae98ee3af620328bb85bbdb5402
I20260925 01:17:15.913787 26211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d: Bootstrap starting.
I20260925 01:17:15.917423 26211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:15.918695 26211 log.cc:824] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:15.922132 26211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d: No bootstrap required, opened a new log
I20260925 01:17:15.922153 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:15.908054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50096 (local address 127.20.102.190:36745)
0925 01:17:15.909155 (+ 1101us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:15.909177 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:15.909230 (+ 53us) server_negotiation.cc:407] Connection header received
0925 01:17:15.911918 (+ 2688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:15.911945 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:15.912511 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:15.912955 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:15.913513 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:15.915562 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:15.916989 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:15.918089 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:15.918271 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:15.919482 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:15.919490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:15.919502 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:15.920106 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:15.920195 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:15.920219 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:15.920361 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:15.920552 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 01:17:15.920887 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":328,"threads_started":1}
I20260925 01:17:15.926194 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:17:15.931577 26211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f29479fd97431fad2b35af0bdb328d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36745 } }
I20260925 01:17:15.931929 26211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:15.932067 26211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11f29479fd97431fad2b35af0bdb328d, State: Initialized, Role: FOLLOWER
I20260925 01:17:15.932601 26211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f29479fd97431fad2b35af0bdb328d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36745 } }
I20260925 01:17:15.932893 26211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:17:15.933050 26211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:17:15.933224 26211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:15.935343 26211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f29479fd97431fad2b35af0bdb328d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36745 } }
I20260925 01:17:15.935817 26211 leader_election.cc:302] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11f29479fd97431fad2b35af0bdb328d; no voters:
I20260925 01:17:15.936384 26211 leader_election.cc:288] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:17:15.936583 26216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:15.937489 26216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [term 1 LEADER]: Becoming Leader. State: Replica: 11f29479fd97431fad2b35af0bdb328d, State: Running, Role: LEADER
I20260925 01:17:15.938058 26216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f29479fd97431fad2b35af0bdb328d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36745 } }
I20260925 01:17:15.938522 26211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:15.941210 26218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11f29479fd97431fad2b35af0bdb328d. Latest consensus state: current_term: 1 leader_uuid: "11f29479fd97431fad2b35af0bdb328d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f29479fd97431fad2b35af0bdb328d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36745 } } }
I20260925 01:17:15.941579 26218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:15.942119 26217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11f29479fd97431fad2b35af0bdb328d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11f29479fd97431fad2b35af0bdb328d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36745 } } }
I20260925 01:17:15.942477 26217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:15.942665 26224 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:17:15.948208 26224 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:17:15.957726 26224 catalog_manager.cc:1347] Generated new cluster ID: 47bef89d33c44fa0865cbb673fc0258a
I20260925 01:17:15.957873 26224 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:17:15.968369 26224 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:17:15.969923 26224 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:17:15.980953 26224 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11f29479fd97431fad2b35af0bdb328d: Generated new TSK 0
I20260925 01:17:15.982021 26224 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 01:17:16.026158 26241 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 01:17:16.026319 26241 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 01:17:16.027593 26214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:16.005391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50100 (local address 127.20.102.190:36745)
0925 01:17:16.005607 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:16.005619 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:16.006183 (+ 564us) server_negotiation.cc:407] Connection header received
0925 01:17:16.006422 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:16.006427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:16.006531 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:16.006717 (+ 186us) server_negotiation.cc:226] Negotiated authn=JWT
0925 01:17:16.007374 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:16.008502 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:16.010142 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:16.010800 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:16.010940 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:16.011555 (+ 615us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 01:17:16.026868 (+ 15313us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 01:17:16.027278 (+ 410us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50100: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 01:17:16.027727 26214 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50100: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 01:17:16.027748 26240 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:16.005217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:36745 (local address 127.0.0.1:50100)
0925 01:17:16.006063 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:16.006092 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:16.006230 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:16.006801 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:16.006808 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:16.006824 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0925 01:17:16.007188 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:16.007200 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:16.008664 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:16.008669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:16.009987 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:16.010002 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:16.011046 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:16.011052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:16.011225 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:16.011263 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 01:17:16.027149 (+ 15886us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 01:17:16.027374 (+ 225us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.102.190:36745: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41875/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":484,"threads_started":1}
I20260925 01:17:16.029188 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 26145
I20260925 01:17:16.030576 26206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:16.127485 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 26145
2026-09-25T01:17:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (724 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 01:17:16.183487 20890 mini_oidc.cc:152] Starting JWKS server
I20260925 01:17:16.185370 20890 webserver.cc:460] Webserver started at https://127.0.0.1:42641/ using document root <none> and password file <none>
I20260925 01:17:16.185588 20890 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 01:17:16.186142 20890 webserver.cc:460] Webserver started at http://127.0.0.1:45371/ using document root <none> and password file <none>
2026-09-25T01:17:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:17:16Z Disabled control of system clock
I20260925 01:17:16.213461 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:35699
--webserver_interface=127.20.102.190
--webserver_port=0
--builtin_ntp_servers=127.20.102.148:45149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:35699
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42641/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 01:17:16.566679 26248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:16.566972 26248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:16.567164 26248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:16.574575 26248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:16.574682 26248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:16.574798 26248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:16.574896 26248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:16.574977 26248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 01:17:16.575062 26248 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 01:17:16.575116 26248 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42641/jwks.json
I20260925 01:17:16.584438 26248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:45149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:35699
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:35699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42641/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:16.585789 26248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:16.587785 26248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:16.599872 26253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:16.599792 26256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:16.599849 26254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:16.601110 26248 server_base.cc:1034] running on GCE node
I20260925 01:17:16.601949 26248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:16.603259 26248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:16.604465 26248 hybrid_clock.cc:648] HybridClock initialized: now 1790299036604404 us; error 47 us; skew 500 ppm
I20260925 01:17:16.604910 26248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:17:16.607215 26248 webserver.cc:460] Webserver started at http://127.20.102.190:41281/ using document root <none> and password file <none>
I20260925 01:17:16.607916 26248 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:16.608036 26248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:16.608477 26248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:16.611487 26248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "93f47a250bc746c1bdd500581e5bad42"
format_stamp: "Formatted at 2026-09-25 01:17:16 on dist-test-slave-s12s"
server_key: "1dd9db23a3f261ecdd24c4ba6152a371"
server_key_iv: "50267fb9672bd14b82e48ee110f7169f"
server_key_version: "encryptionkey@0"
I20260925 01:17:16.612308 26248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "93f47a250bc746c1bdd500581e5bad42"
format_stamp: "Formatted at 2026-09-25 01:17:16 on dist-test-slave-s12s"
server_key: "1dd9db23a3f261ecdd24c4ba6152a371"
server_key_iv: "50267fb9672bd14b82e48ee110f7169f"
server_key_version: "encryptionkey@0"
I20260925 01:17:16.617285 26248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 01:17:16.620472 26262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:16.621915 26248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:16.622179 26248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "93f47a250bc746c1bdd500581e5bad42"
format_stamp: "Formatted at 2026-09-25 01:17:16 on dist-test-slave-s12s"
server_key: "1dd9db23a3f261ecdd24c4ba6152a371"
server_key_iv: "50267fb9672bd14b82e48ee110f7169f"
server_key_version: "encryptionkey@0"
I20260925 01:17:16.622376 26248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:16.640784 26248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:16.641525 26248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:16.641789 26248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:16.660341 26248 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:35699
I20260925 01:17:16.660364 26313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:35699 every 8 connection(s)
I20260925 01:17:16.661724 26248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:17:16.663177 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 26248
I20260925 01:17:16.663417 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:17:16.664300 20890 external_mini_cluster.cc:1428] Setting key 37f3f10989d84bc6f70eee904b78895b
I20260925 01:17:16.666733 26314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:16.679005 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:16.666400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44138 (local address 127.20.102.190:35699)
0925 01:17:16.667456 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:16.667471 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:16.667515 (+ 44us) server_negotiation.cc:407] Connection header received
0925 01:17:16.669221 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:16.669260 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:16.669703 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:16.670388 (+ 685us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:16.671089 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:16.672880 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:16.674309 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:16.675026 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:16.675255 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:16.676521 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:16.676529 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:16.676542 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:16.677144 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:16.677216 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:16.677237 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:16.677364 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:16.677620 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 01:17:16.678000 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":438,"threads_started":1}
I20260925 01:17:16.682956 26314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42: Bootstrap starting.
I20260925 01:17:16.683442 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:17:16.687220 26314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:16.688416 26314 log.cc:824] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:16.691354 26314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42: No bootstrap required, opened a new log
I20260925 01:17:16.699131 26314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f47a250bc746c1bdd500581e5bad42" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35699 } }
I20260925 01:17:16.699497 26314 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:16.699632 26314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f47a250bc746c1bdd500581e5bad42, State: Initialized, Role: FOLLOWER
I20260925 01:17:16.700274 26314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f47a250bc746c1bdd500581e5bad42" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35699 } }
I20260925 01:17:16.700515 26314 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:17:16.700673 26314 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:17:16.700892 26314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:16.703280 26314 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f47a250bc746c1bdd500581e5bad42" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35699 } }
I20260925 01:17:16.703799 26314 leader_election.cc:302] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93f47a250bc746c1bdd500581e5bad42; no voters:
I20260925 01:17:16.704414 26314 leader_election.cc:288] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:17:16.704591 26319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:16.705317 26319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [term 1 LEADER]: Becoming Leader. State: Replica: 93f47a250bc746c1bdd500581e5bad42, State: Running, Role: LEADER
I20260925 01:17:16.705808 26319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f47a250bc746c1bdd500581e5bad42" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35699 } }
I20260925 01:17:16.706612 26314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:16.708798 26320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93f47a250bc746c1bdd500581e5bad42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f47a250bc746c1bdd500581e5bad42" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35699 } } }
I20260925 01:17:16.709242 26320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:16.708972 26321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93f47a250bc746c1bdd500581e5bad42. Latest consensus state: current_term: 1 leader_uuid: "93f47a250bc746c1bdd500581e5bad42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f47a250bc746c1bdd500581e5bad42" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35699 } } }
I20260925 01:17:16.709432 26321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:16.711840 26325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:17:16.717550 26325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:17:16.725407 26325 catalog_manager.cc:1347] Generated new cluster ID: da6ed89fc07a4415845210cd1a6d5d08
I20260925 01:17:16.725514 26325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:17:16.745402 26325 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:17:16.746663 26325 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:17:16.752264 26325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93f47a250bc746c1bdd500581e5bad42: Generated new TSK 0
I20260925 01:17:16.753180 26325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:17:16.775015 26317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:16.766205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44152 (local address 127.20.102.190:35699)
0925 01:17:16.766432 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:16.766437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:16.766820 (+ 383us) server_negotiation.cc:407] Connection header received
0925 01:17:16.767082 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:16.767086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:16.767153 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:16.767268 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0925 01:17:16.767915 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:16.768853 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:16.771043 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:16.771525 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:16.771634 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:16.772294 (+ 660us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 01:17:16.774318 (+ 2024us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42641/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 01:17:16.774699 (+ 381us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44152: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42641/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 01:17:16.775249 26343 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:16.766049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35699 (local address 127.0.0.1:44152)
0925 01:17:16.766708 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:16.766729 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:16.766887 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:16.767324 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:16.767331 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:16.767349 (+ 18us) client_negotiation.cc:189] Negotiated authn=JWT
0925 01:17:16.767714 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:16.767730 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:16.769014 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:16.769020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:16.770869 (+ 1849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:16.770886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:16.771737 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:16.771745 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:16.771956 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:16.772008 (+ 52us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 01:17:16.774545 (+ 2537us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42641/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 01:17:16.774778 (+ 233us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.102.190:35699: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42641/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":338,"threads_started":1}
W20260925 01:17:16.775547 26317 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44152: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42641/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260925 01:17:16.776988 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 26248
I20260925 01:17:16.778311 26309 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 01:17:16.789183 26344 webserver.cc:533] Webserver: sslize failed:
E20260925 01:17:16.789275 26344 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 01:17:16.868767 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 26248
2026-09-25T01:17:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (741 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T01:17:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:17:16Z Disabled control of system clock
I20260925 01:17:16.953786 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:45013
--webserver_interface=127.20.102.190
--webserver_port=0
--builtin_ntp_servers=127.20.102.148:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:45013
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/testchainca.cert with env {}
W20260925 01:17:17.277405 26349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:17.277773 26349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:17.277956 26349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:17.285868 26349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:17.286019 26349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:17.286118 26349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:17.286222 26349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:17.286300 26349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 01:17:17.286388 26349 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 01:17:17.286469 26349 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 01:17:17.297089 26349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:45013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:45013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:17.298457 26349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:17.300266 26349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:17.312359 26357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:17.312400 26354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:17.312433 26355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:17.318117 26349 server_base.cc:1034] running on GCE node
I20260925 01:17:17.319008 26349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:17.320585 26349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:17.321869 26349 hybrid_clock.cc:648] HybridClock initialized: now 1790299037321807 us; error 65 us; skew 500 ppm
I20260925 01:17:17.322263 26349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:17:17.324813 26349 webserver.cc:460] Webserver started at http://127.20.102.190:40215/ using document root <none> and password file <none>
I20260925 01:17:17.325543 26349 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:17.325686 26349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:17.326032 26349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:17.329087 26349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "93174de6fa114e10b3135ca45ffa935a"
format_stamp: "Formatted at 2026-09-25 01:17:17 on dist-test-slave-s12s"
server_key: "83590f7577e965a9a4cd6137b8f7764e"
server_key_iv: "ae906b2a891cda0c87d0571b0370670a"
server_key_version: "encryptionkey@0"
I20260925 01:17:17.329945 26349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "93174de6fa114e10b3135ca45ffa935a"
format_stamp: "Formatted at 2026-09-25 01:17:17 on dist-test-slave-s12s"
server_key: "83590f7577e965a9a4cd6137b8f7764e"
server_key_iv: "ae906b2a891cda0c87d0571b0370670a"
server_key_version: "encryptionkey@0"
I20260925 01:17:17.335371 26349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:17:17.338721 26363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:17.340005 26349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:17:17.340184 26349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "93174de6fa114e10b3135ca45ffa935a"
format_stamp: "Formatted at 2026-09-25 01:17:17 on dist-test-slave-s12s"
server_key: "83590f7577e965a9a4cd6137b8f7764e"
server_key_iv: "ae906b2a891cda0c87d0571b0370670a"
server_key_version: "encryptionkey@0"
I20260925 01:17:17.340394 26349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:17.354277 26349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:17.355054 26349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:17.355343 26349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:17.373798 26349 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:45013
I20260925 01:17:17.373818 26414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:45013 every 8 connection(s)
I20260925 01:17:17.375200 26349 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:17:17.378882 26415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:17.381949 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 26349
I20260925 01:17:17.382154 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:17:17.383010 20890 external_mini_cluster.cc:1428] Setting key a973255f5dc34f838ee74b1d92dd5c64
I20260925 01:17:17.391270 26415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a: Bootstrap starting.
I20260925 01:17:17.394403 26415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:17.395553 26415 log.cc:824] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:17.396531 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:17.385059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.20.102.190:45013)
0925 01:17:17.386049 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:17.386059 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:17.386101 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:17:17.387714 (+ 1613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:17.387737 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:17.388238 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:17.388658 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:17.389267 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:17.390521 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:17.391866 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:17.392530 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:17.392769 (+ 239us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:17.393932 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:17.393940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:17.393954 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:17:17.394540 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:17.394649 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:17:17.394671 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:17.394798 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:17.395033 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 01:17:17.395341 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":359,"threads_started":1}
I20260925 01:17:17.398801 26415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a: No bootstrap required, opened a new log
I20260925 01:17:17.400329 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:17:17.406634 26415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93174de6fa114e10b3135ca45ffa935a" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45013 } }
I20260925 01:17:17.407004 26415 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:17.407122 26415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93174de6fa114e10b3135ca45ffa935a, State: Initialized, Role: FOLLOWER
I20260925 01:17:17.407694 26415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93174de6fa114e10b3135ca45ffa935a" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45013 } }
I20260925 01:17:17.407925 26415 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:17:17.408070 26415 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:17:17.408232 26415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:17.410502 26415 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93174de6fa114e10b3135ca45ffa935a" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45013 } }
I20260925 01:17:17.410961 26415 leader_election.cc:302] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93174de6fa114e10b3135ca45ffa935a; no voters:
I20260925 01:17:17.411573 26415 leader_election.cc:288] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:17:17.411755 26420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:17.412536 26420 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [term 1 LEADER]: Becoming Leader. State: Replica: 93174de6fa114e10b3135ca45ffa935a, State: Running, Role: LEADER
I20260925 01:17:17.413215 26420 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93174de6fa114e10b3135ca45ffa935a" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45013 } }
I20260925 01:17:17.413625 26415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:17.416033 26421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93174de6fa114e10b3135ca45ffa935a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93174de6fa114e10b3135ca45ffa935a" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45013 } } }
I20260925 01:17:17.416467 26421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:17.417032 26422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93174de6fa114e10b3135ca45ffa935a. Latest consensus state: current_term: 1 leader_uuid: "93174de6fa114e10b3135ca45ffa935a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93174de6fa114e10b3135ca45ffa935a" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 45013 } } }
I20260925 01:17:17.417389 26422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:17.418038 26429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:17:17.423573 26429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:17:17.432811 26429 catalog_manager.cc:1347] Generated new cluster ID: 43fe362cabb844ac9e70f80a6ff57e16
I20260925 01:17:17.432924 26429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:17:17.477488 26429 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:17:17.478868 26429 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:17:17.495366 26429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93174de6fa114e10b3135ca45ffa935a: Generated new TSK 0
I20260925 01:17:17.496234 26429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:17:17.566107 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:17.519553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41100 (local address 127.20.102.190:45013)
0925 01:17:17.519715 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:17.519719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:17.520214 (+ 495us) server_negotiation.cc:407] Connection header received
0925 01:17:17.520456 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:17.520460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:17.520530 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:17.520731 (+ 201us) server_negotiation.cc:226] Negotiated authn=JWT
0925 01:17:17.521193 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:17.521912 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:17.523287 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:17.523745 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:17.523884 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:17.524445 (+ 561us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 01:17:17.565338 (+ 40893us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 01:17:17.565773 (+ 435us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41100: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 01:17:17.566268 26418 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41100: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 01:17:17.566383 26444 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:17.519556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:45013 (local address 127.0.0.1:41100)
0925 01:17:17.520112 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:17.520128 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:17.520285 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:17.520747 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:17.520752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:17.520764 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0925 01:17:17.521028 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:17.521037 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:17.522074 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:17.522078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:17.523145 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:17.523156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:17.523943 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:17.523987 (+ 44us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:17.524165 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:17.524202 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 01:17:17.565634 (+ 41432us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 01:17:17.565937 (+ 303us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.20.102.190:45013: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":284,"threads_started":1}
I20260925 01:17:17.567119 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 26349
I20260925 01:17:17.568337 26410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:17.670346 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 26349
2026-09-25T01:17:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (781 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (412 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (412 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:17:18 dist-test-slave-s12s krb5kdc[26459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:17:18 dist-test-slave-s12s krb5kdc[26459](info): set up 2 sockets
Sep 25 01:17:18 dist-test-slave-s12s krb5kdc[26459](info): commencing operation
krb5kdc: starting...
W20260925 01:17:19.581449 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:17:19 dist-test-slave-s12s krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:17:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:17:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:17:19.789542 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:37731
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:37731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:20.106290 26472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:20.106597 26472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:20.106753 26472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:20.113673 26472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:20.113806 26472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:20.113906 26472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:20.114003 26472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:20.114087 26472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:17:20.123395 26472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:37731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:37731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:20.124703 26472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:20.125976 26472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:20.137993 26478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:20.138132 26480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:20.138200 26477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:20.139110 26472 server_base.cc:1034] running on GCE node
I20260925 01:17:20.139863 26472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:20.141328 26472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:20.142575 26472 hybrid_clock.cc:648] HybridClock initialized: now 1790299040142523 us; error 56 us; skew 500 ppm
I20260925 01:17:20.142922 26472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:20 dist-test-slave-s12s krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299040, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:20.149632 26472 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:17:20.151103 26472 webserver.cc:460] Webserver started at http://127.20.102.190:41901/ using document root <none> and password file <none>
I20260925 01:17:20.151798 26472 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:20.151932 26472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:20.152235 26472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:20.155246 26472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "da9fb0bb4a224b1fbdf55a82b25f65b9"
format_stamp: "Formatted at 2026-09-25 01:17:20 on dist-test-slave-s12s"
server_key: "956725c49cf4761ce75fe1ac0debf296"
server_key_iv: "e2406183c835003d885386a838988808"
server_key_version: "encryptionkey@0"
I20260925 01:17:20.155997 26472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "da9fb0bb4a224b1fbdf55a82b25f65b9"
format_stamp: "Formatted at 2026-09-25 01:17:20 on dist-test-slave-s12s"
server_key: "956725c49cf4761ce75fe1ac0debf296"
server_key_iv: "e2406183c835003d885386a838988808"
server_key_version: "encryptionkey@0"
I20260925 01:17:20.161058 26472 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 01:17:20.164235 26487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:20.165575 26472 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:20.165743 26472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "da9fb0bb4a224b1fbdf55a82b25f65b9"
format_stamp: "Formatted at 2026-09-25 01:17:20 on dist-test-slave-s12s"
server_key: "956725c49cf4761ce75fe1ac0debf296"
server_key_iv: "e2406183c835003d885386a838988808"
server_key_version: "encryptionkey@0"
I20260925 01:17:20.165928 26472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:20.179637 26472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:20.183791 26472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:20.184073 26472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:20.202075 26472 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:37731
I20260925 01:17:20.202104 26538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:37731 every 8 connection(s)
I20260925 01:17:20.203390 26472 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:17:20.206310 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 26472
I20260925 01:17:20.206527 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:17:20.207356 20890 external_mini_cluster.cc:1428] Setting key bf4d0feeb6de5c36cd75cb8627c1d8bc
I20260925 01:17:20.207520 26539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:20.219735 26539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9: Bootstrap starting.
Sep 25 01:17:20 dist-test-slave-s12s krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:20.222856 26539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:20.223884 26539 log.cc:824] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:20.226549 26539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9: No bootstrap required, opened a new log
I20260925 01:17:20.229434 26542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:20.209449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.20.102.190:37731)
0925 01:17:20.210183 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:20.210194 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:20.210235 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:17:20.212234 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:20.212255 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:20.212616 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:20.213068 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:20.214458 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:20.215706 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:20.217017 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:20.217713 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:20.217942 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:20.222000 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:20.222007 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:20.222019 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:20.222057 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:20.225533 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:20.226187 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:20.226193 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:20.226198 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:20.226348 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:20.226768 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:20.226771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:20.226773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:20.227272 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:20.227590 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:20.228039 (+ 449us) server_negotiation.cc:299] Negotiation successful
0925 01:17:20.228439 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":320,"threads_started":1}
I20260925 01:17:20.234344 26539 raft_consensus.cc:348] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9fb0bb4a224b1fbdf55a82b25f65b9" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37731 } }
I20260925 01:17:20.234681 26539 raft_consensus.cc:374] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:20.234790 26539 raft_consensus.cc:729] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da9fb0bb4a224b1fbdf55a82b25f65b9, State: Initialized, Role: FOLLOWER
I20260925 01:17:20.235399 26539 consensus_queue.cc:260] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9fb0bb4a224b1fbdf55a82b25f65b9" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37731 } }
I20260925 01:17:20.235705 26539 raft_consensus.cc:388] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:17:20.235862 26539 raft_consensus.cc:482] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:17:20.236068 26539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:20.238183 26539 raft_consensus.cc:504] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9fb0bb4a224b1fbdf55a82b25f65b9" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37731 } }
I20260925 01:17:20.238636 26539 leader_election.cc:302] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da9fb0bb4a224b1fbdf55a82b25f65b9; no voters:
I20260925 01:17:20.239163 26539 leader_election.cc:288] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:17:20.239272 26544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:20.239961 26544 raft_consensus.cc:686] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [term 1 LEADER]: Becoming Leader. State: Replica: da9fb0bb4a224b1fbdf55a82b25f65b9, State: Running, Role: LEADER
I20260925 01:17:20.240505 26544 consensus_queue.cc:237] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9fb0bb4a224b1fbdf55a82b25f65b9" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37731 } }
I20260925 01:17:20.241237 26539 sys_catalog.cc:564] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:20.243384 26545 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da9fb0bb4a224b1fbdf55a82b25f65b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9fb0bb4a224b1fbdf55a82b25f65b9" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37731 } } }
I20260925 01:17:20.243562 26546 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da9fb0bb4a224b1fbdf55a82b25f65b9. Latest consensus state: current_term: 1 leader_uuid: "da9fb0bb4a224b1fbdf55a82b25f65b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9fb0bb4a224b1fbdf55a82b25f65b9" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37731 } } }
I20260925 01:17:20.243769 26545 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:20.243825 26546 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:20.246286 26552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:17:20.251539 26552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:17:20.260265 26552 catalog_manager.cc:1347] Generated new cluster ID: ed848bbd275a40899e9800d7b4a5e3b5
I20260925 01:17:20.260376 26552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:17:20.275408 26552 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:17:20.276607 26552 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:17:20.283720 26552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da9fb0bb4a224b1fbdf55a82b25f65b9: Generated new TSK 0
I20260925 01:17:20.284430 26552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:17:20.336100 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37731
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:20.654373 26565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:20.654627 26565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:20.654779 26565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:20.661800 26565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:20.661909 26565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:20.662074 26565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:17:20.671504 26565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:20.672780 26565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:20.674094 26565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:20.686712 26570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:20.686698 26573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:20.686743 26571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:20.687708 26565 server_base.cc:1034] running on GCE node
I20260925 01:17:20.688284 26565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:20.689183 26565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:20.690367 26565 hybrid_clock.cc:648] HybridClock initialized: now 1790299040690318 us; error 54 us; skew 500 ppm
I20260925 01:17:20.690624 26565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:20 dist-test-slave-s12s krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299040, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:20.693535 26565 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:17:20.694908 26565 webserver.cc:460] Webserver started at http://127.20.102.129:35793/ using document root <none> and password file <none>
I20260925 01:17:20.695566 26565 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:20.695703 26565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:20.695993 26565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:20.698810 26565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "17f8a99ddaef42d081866f9c36e6898e"
format_stamp: "Formatted at 2026-09-25 01:17:20 on dist-test-slave-s12s"
server_key: "ccffeb9a7e765982d7cdfa69a00b5328"
server_key_iv: "1d4de331c9113565e3caa441bd70ed0f"
server_key_version: "encryptionkey@0"
I20260925 01:17:20.699542 26565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "17f8a99ddaef42d081866f9c36e6898e"
format_stamp: "Formatted at 2026-09-25 01:17:20 on dist-test-slave-s12s"
server_key: "ccffeb9a7e765982d7cdfa69a00b5328"
server_key_iv: "1d4de331c9113565e3caa441bd70ed0f"
server_key_version: "encryptionkey@0"
I20260925 01:17:20.704380 26565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 01:17:20.708206 26581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:20.709671 26565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:20.709856 26565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "17f8a99ddaef42d081866f9c36e6898e"
format_stamp: "Formatted at 2026-09-25 01:17:20 on dist-test-slave-s12s"
server_key: "ccffeb9a7e765982d7cdfa69a00b5328"
server_key_iv: "1d4de331c9113565e3caa441bd70ed0f"
server_key_version: "encryptionkey@0"
I20260925 01:17:20.710076 26565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:20.723255 26565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:20.727133 26565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:20.727412 26565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:20.728299 26565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:20.729588 26565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:20.729703 26565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:20.729859 26565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:20.729938 26565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:20.765210 26565 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:37399
I20260925 01:17:20.765255 26693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:37399 every 8 connection(s)
I20260925 01:17:20.766494 26565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:17:20.774019 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 26565
I20260925 01:17:20.774230 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:17:20.774906 20890 external_mini_cluster.cc:1428] Setting key e6d5c1b0545c73a8fde7d0438a217902
Sep 25 01:17:20 dist-test-slave-s12s krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299040, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:20.787317 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:20.769166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:54195 (local address 127.20.102.190:37731)
0925 01:17:20.769657 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:20.769661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:20.770594 (+ 933us) server_negotiation.cc:407] Connection header received
0925 01:17:20.772459 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:20.772465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:20.772563 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:20.772680 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:20.774203 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:20.775029 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:20.776854 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:20.777263 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:20.777394 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:20.781048 (+ 3654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:20.781053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:20.781056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:20.781087 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:20.784037 (+ 2950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:20.784741 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:20.784750 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:20.784757 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:20.784905 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:20.785332 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:20.785339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:20.785346 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:20.785779 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:20.785940 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:20.786591 (+ 651us) server_negotiation.cc:299] Negotiation successful
0925 01:17:20.786834 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":281,"threads_started":1}
I20260925 01:17:20.787920 26697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:20.769715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37731 (local address 127.20.102.129:54195)
0925 01:17:20.770412 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:20.770477 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:20.772190 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:20.772940 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:20.772949 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:20.773467 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:20.774025 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:20.774052 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:20.775171 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:20.775183 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:20.776668 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:20.776722 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:20.777483 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:20.777493 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:20.777740 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:20.778433 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:20.778454 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:20.780839 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:20.784178 (+ 3339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:20.784185 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:20.784197 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:20.784594 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:20.785023 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:20.785026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:20.785028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:20.785216 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:20.785908 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:20.785914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:20.786239 (+ 325us) client_negotiation.cc:769] Sending connection context
0925 01:17:20.786446 (+ 207us) client_negotiation.cc:240] Negotiation successful
0925 01:17:20.786813 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":274,"threads_started":1}
I20260925 01:17:20.789537 26694 heartbeater.cc:347] Connected to a master server at 127.20.102.190:37731
I20260925 01:17:20.789881 26694 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:20.790752 26694 heartbeater.cc:511] Master 127.20.102.190:37731 requested a full tablet report, sending...
I20260925 01:17:20.793295 26504 ts_manager.cc:194] Registered new tserver with Master: 17f8a99ddaef42d081866f9c36e6898e (127.20.102.129:37399)
I20260925 01:17:20.795549 26504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:54195
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:17:20.823181 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37731
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:21.145900 26700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:21.146180 26700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:21.146274 26700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:21.153602 26700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:21.153698 26700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:21.153810 26700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:17:21.163194 26700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:21.164407 26700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:21.165691 26700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:21.178532 26708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:21.178622 26705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:21.178565 26706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:21.179412 26700 server_base.cc:1034] running on GCE node
I20260925 01:17:21.180015 26700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:21.180904 26700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:21.182104 26700 hybrid_clock.cc:648] HybridClock initialized: now 1790299041182066 us; error 34 us; skew 500 ppm
I20260925 01:17:21.182422 26700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:21 dist-test-slave-s12s krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:21.185431 26700 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:17:21.186766 26700 webserver.cc:460] Webserver started at http://127.20.102.130:39967/ using document root <none> and password file <none>
I20260925 01:17:21.187427 26700 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:21.187572 26700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:21.187870 26700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:21.190835 26700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "c7b04c776b47449aa875397d940bef46"
format_stamp: "Formatted at 2026-09-25 01:17:21 on dist-test-slave-s12s"
server_key: "1ac4047c7c31f60d7ea00d3d4f0088cd"
server_key_iv: "f955c188870b8250daa61d3afada6aeb"
server_key_version: "encryptionkey@0"
I20260925 01:17:21.191602 26700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "c7b04c776b47449aa875397d940bef46"
format_stamp: "Formatted at 2026-09-25 01:17:21 on dist-test-slave-s12s"
server_key: "1ac4047c7c31f60d7ea00d3d4f0088cd"
server_key_iv: "f955c188870b8250daa61d3afada6aeb"
server_key_version: "encryptionkey@0"
I20260925 01:17:21.196614 26700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:17:21.199962 26715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:21.201339 26700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:21.201499 26700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "c7b04c776b47449aa875397d940bef46"
format_stamp: "Formatted at 2026-09-25 01:17:21 on dist-test-slave-s12s"
server_key: "1ac4047c7c31f60d7ea00d3d4f0088cd"
server_key_iv: "f955c188870b8250daa61d3afada6aeb"
server_key_version: "encryptionkey@0"
I20260925 01:17:21.201713 26700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:21.236901 26700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:21.241933 26700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:21.242214 26700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:21.243263 26700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:21.244823 26700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:21.244897 26700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:21.244992 26700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:21.245034 26700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:21.277689 26700 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:41745
I20260925 01:17:21.277746 26827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:41745 every 8 connection(s)
I20260925 01:17:21.279035 26700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:17:21.281440 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 26700
I20260925 01:17:21.281641 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:17:21.282168 20890 external_mini_cluster.cc:1428] Setting key 30ee2e56561bdc27548a2717652aa2e7
Sep 25 01:17:21 dist-test-slave-s12s krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:21.302874 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:21.281795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:47309 (local address 127.20.102.190:37731)
0925 01:17:21.282006 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:21.282013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:21.283400 (+ 1387us) server_negotiation.cc:407] Connection header received
0925 01:17:21.285783 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:21.285794 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:21.285859 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:21.285937 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:21.288017 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.288911 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.291420 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.292152 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.292303 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:21.295573 (+ 3270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:21.295582 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:21.295589 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:21.295633 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:21.299325 (+ 3692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:21.300126 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:21.300135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:21.300141 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:21.300315 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:21.300822 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:21.300828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:21.300833 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:21.301243 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:21.301405 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:21.302155 (+ 750us) server_negotiation.cc:299] Negotiation successful
0925 01:17:21.302440 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 01:17:21.303531 26830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:21.282424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37731 (local address 127.20.102.130:47309)
0925 01:17:21.283251 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:21.283290 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:21.285477 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:21.286230 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:21.286243 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:21.287030 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:21.287837 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:21.287865 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.289081 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.289088 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:21.291250 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:21.291267 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.292354 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.292359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:21.292545 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:21.293242 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:21.293268 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:21.295358 (+ 2090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:21.299540 (+ 4182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:21.299548 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:21.299561 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:21.299992 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:21.300429 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:21.300432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:21.300435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:21.300643 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:21.301371 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:21.301377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:21.301742 (+ 365us) client_negotiation.cc:769] Sending connection context
0925 01:17:21.301978 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 01:17:21.302382 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":321,"threads_started":1}
I20260925 01:17:21.305056 26828 heartbeater.cc:347] Connected to a master server at 127.20.102.190:37731
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
I20260925 01:17:21.305392 26828 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:21.306156 26828 heartbeater.cc:511] Master 127.20.102.190:37731 requested a full tablet report, sending...
I20260925 01:17:21.307654 26504 ts_manager.cc:194] Registered new tserver with Master: c7b04c776b47449aa875397d940bef46 (127.20.102.130:41745)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
I20260925 01:17:21.308648 26504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:47309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:17:21.332201 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37731
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:21.654505 26833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:21.654769 26833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:21.654942 26833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:21.662106 26833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:21.662242 26833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:21.662416 26833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:17:21.672117 26833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:21.673363 26833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:21.674646 26833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:21.688306 26839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:21.689041 26838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:21.688482 26841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:21.691184 26833 server_base.cc:1034] running on GCE node
I20260925 01:17:21.691831 26833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:21.692762 26833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:21.693951 26833 hybrid_clock.cc:648] HybridClock initialized: now 1790299041693896 us; error 47 us; skew 500 ppm
I20260925 01:17:21.694214 26833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:21 dist-test-slave-s12s krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:21.697265 26833 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:17:21.698781 26833 webserver.cc:460] Webserver started at http://127.20.102.131:40365/ using document root <none> and password file <none>
I20260925 01:17:21.699469 26833 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:21.699606 26833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:21.699908 26833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:21.702783 26833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "c844b437f717440cad5cf2ab2fc5d40c"
format_stamp: "Formatted at 2026-09-25 01:17:21 on dist-test-slave-s12s"
server_key: "2bdda83e5391286e204615d5b199985e"
server_key_iv: "703e3e03a7c1366c1bdb89e424fb714d"
server_key_version: "encryptionkey@0"
I20260925 01:17:21.703548 26833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "c844b437f717440cad5cf2ab2fc5d40c"
format_stamp: "Formatted at 2026-09-25 01:17:21 on dist-test-slave-s12s"
server_key: "2bdda83e5391286e204615d5b199985e"
server_key_iv: "703e3e03a7c1366c1bdb89e424fb714d"
server_key_version: "encryptionkey@0"
I20260925 01:17:21.708650 26833 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 01:17:21.712049 26848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:21.713507 26833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:21.713682 26833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "c844b437f717440cad5cf2ab2fc5d40c"
format_stamp: "Formatted at 2026-09-25 01:17:21 on dist-test-slave-s12s"
server_key: "2bdda83e5391286e204615d5b199985e"
server_key_iv: "703e3e03a7c1366c1bdb89e424fb714d"
server_key_version: "encryptionkey@0"
I20260925 01:17:21.713850 26833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:21.731937 26833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:21.736011 26833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:21.736279 26833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:21.737203 26833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:21.738492 26833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:21.738564 26833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:21.738722 26833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:21.738775 26833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:21.773669 26833 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:35105
I20260925 01:17:21.773705 26960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:35105 every 8 connection(s)
I20260925 01:17:21.774926 26833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:17:21.781337 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 26833
I20260925 01:17:21.781533 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:17:21.782094 20890 external_mini_cluster.cc:1428] Setting key 01f7821479bb02440a6c3fff9bb3b274
Sep 25 01:17:21 dist-test-slave-s12s krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:21.794772 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:21.777487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:44151 (local address 127.20.102.190:37731)
0925 01:17:21.777712 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:21.777716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:21.778894 (+ 1178us) server_negotiation.cc:407] Connection header received
0925 01:17:21.780756 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:21.780763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:21.780851 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:21.780940 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:21.782972 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.783922 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.785667 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.786069 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.786166 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:21.789785 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:21.789790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:21.789793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:21.789827 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:21.792149 (+ 2322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:21.792896 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:21.792900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:21.792902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:21.793025 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:21.793400 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:21.793403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:21.793405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:21.793662 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:21.793769 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:21.794381 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 01:17:21.794529 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260925 01:17:21.795840 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:21.777981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37731 (local address 127.20.102.131:44151)
0925 01:17:21.778725 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:21.778778 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:21.780415 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:21.781221 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:21.781233 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:21.782006 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:21.782770 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:21.782809 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.784051 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.784058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:21.785548 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:21.785559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.786296 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.786301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:21.786541 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:21.787279 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:21.787309 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:21.789588 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:21.792271 (+ 2683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:21.792277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:21.792288 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:21.792800 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:21.793119 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:21.793123 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:21.793125 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:21.793306 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:21.793768 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:21.793774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:21.794081 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 01:17:21.794310 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 01:17:21.794696 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":298,"threads_started":1}
I20260925 01:17:21.797326 26961 heartbeater.cc:347] Connected to a master server at 127.20.102.190:37731
I20260925 01:17:21.797639 26961 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:21.798369 26961 heartbeater.cc:511] Master 127.20.102.190:37731 requested a full tablet report, sending...
I20260925 01:17:21.798789 26694 heartbeater.cc:503] Master 127.20.102.190:37731 was elected leader, sending a full tablet report...
I20260925 01:17:21.799846 26504 ts_manager.cc:194] Registered new tserver with Master: c844b437f717440cad5cf2ab2fc5d40c (127.20.102.131:35105)
I20260925 01:17:21.800734 26504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:44151
I20260925 01:17:21.809440 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/test-admin.keytab.
Sep 25 01:17:21 dist-test-slave-s12s krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:21.835837 20890 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 01:17:21.841154 26968 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:21.838533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.20.102.129:37399)
0925 01:17:21.839214 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:21.839222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:21.839251 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:17:21.839314 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:21.839321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:21.839553 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:21.839688 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:21.840648 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51478: BlockingRecv error: recv got EOF from 127.0.0.1:51478 (error 108)
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":373,"threads_started":1}
W20260925 01:17:21.841177 20890 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/krb5cc' of type FILE
Sep 25 01:17:21 dist-test-slave-s12s krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:21.842900 20890 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 01:17:21 dist-test-slave-s12s krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:21.844161 20890 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 01:17:21 dist-test-slave-s12s krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:17:21.859437 26968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:21.845750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.20.102.129:37399)
0925 01:17:21.845941 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:21.845947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:21.846141 (+ 194us) server_negotiation.cc:407] Connection header received
0925 01:17:21.846325 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:21.846335 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:21.846392 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:21.846489 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:21.847482 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.848424 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.849701 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.850254 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.850399 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:21.853802 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:21.853813 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:21.853827 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:21.853861 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:21.856879 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:21.857696 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:21.857703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:21.857705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:21.857827 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:21.858144 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:21.858148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:21.858150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:21.858530 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:21.858758 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:21.858974 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 01:17:21.859179 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 01:17:21.865072 26968 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:21.863606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.20.102.129:37399)
0925 01:17:21.863780 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:21.863784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:21.863882 (+ 98us) server_negotiation.cc:407] Connection header received
0925 01:17:21.864018 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:21.864022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:21.864084 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:21.864157 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:21.864847 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44964: BlockingRecv error: recv got EOF from 127.0.0.1:44964 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 01:17:21.865252 20890 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/krb5cc' of type FILE
Sep 25 01:17:21 dist-test-slave-s12s krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:21.866734 20890 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 01:17:21 dist-test-slave-s12s krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:21.868093 20890 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 01:17:21 dist-test-slave-s12s krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:17:21.883035 26968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:21.869384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44978 (local address 127.20.102.129:37399)
0925 01:17:21.869621 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:21.869627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:21.869882 (+ 255us) server_negotiation.cc:407] Connection header received
0925 01:17:21.870065 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:21.870071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:21.870141 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:21.870242 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:21.871253 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.872026 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.873412 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.873886 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.874030 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:21.877613 (+ 3583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:21.877619 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:21.877622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:21.877669 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:21.880202 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:21.880923 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:21.880930 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:21.880934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:21.881076 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:21.881518 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:21.881523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:21.881525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:21.881790 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:21.882074 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:21.882388 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 01:17:21.882616 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260925 01:17:21.887109 26968 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:21.885579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.20.102.129:37399)
0925 01:17:21.885771 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:21.885777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:21.885936 (+ 159us) server_negotiation.cc:407] Connection header received
0925 01:17:21.886068 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:21.886071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:21.886134 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:21.886208 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:21.886920 (+ 712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44992: BlockingRecv error: recv got EOF from 127.0.0.1:44992 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 01:17:21.887313 20890 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790298989998230-20890-0/krb5kdc/krb5cc' of type FILE
Sep 25 01:17:21 dist-test-slave-s12s krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:21.888864 20890 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 01:17:21 dist-test-slave-s12s krb5kdc[26459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:21.890197 20890 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 01:17:21 dist-test-slave-s12s krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:17:21.904111 26968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:21.891414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44996 (local address 127.20.102.129:37399)
0925 01:17:21.891619 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:21.891625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:21.891897 (+ 272us) server_negotiation.cc:407] Connection header received
0925 01:17:21.892103 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:21.892107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:21.892162 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:21.892262 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:21.893340 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.894100 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.895428 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.895909 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.896065 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:21.899457 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:21.899462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:21.899466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:21.899498 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:21.902023 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:21.902573 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:21.902577 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:21.902579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:21.902723 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:21.903032 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:21.903035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:21.903037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:21.903271 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:21.903487 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:21.903682 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 01:17:21.903839 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:17:21.905678 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 26565
I20260925 01:17:21.917448 26968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:21.906361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.20.102.129:37399)
0925 01:17:21.906526 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:21.906530 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:21.906895 (+ 365us) server_negotiation.cc:407] Connection header received
0925 01:17:21.907047 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:21.907053 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:21.907113 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:21.907216 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:21.908207 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.908919 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.910065 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:21.910485 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:21.910615 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:21.912487 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:21.912493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:21.912498 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:21.912531 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:21.914958 (+ 2427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:21.915581 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:21.915587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:21.915590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:21.915716 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:21.916095 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:21.916100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:21.916107 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:21.916333 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:21.916585 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:21.916930 (+ 345us) server_negotiation.cc:299] Negotiation successful
0925 01:17:21.917136 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 01:17:21.917985 26689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:22.008168 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 26565
I20260925 01:17:22.025496 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 26700
Sep 25 01:17:22 dist-test-slave-s12s krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:17:22.040373 26981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:22.026464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45574 (local address 127.20.102.130:41745)
0925 01:17:22.027067 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:22.027075 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:22.027103 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:17:22.027171 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:22.027177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:22.027394 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:22.027544 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:22.028465 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:22.029573 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:22.030768 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:22.031310 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:22.031409 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:22.035141 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:22.035152 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:22.035166 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:22.035210 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:22.038018 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:22.038578 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:22.038585 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:22.038587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:22.038711 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:22.038991 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:22.038995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:22.038998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:22.039305 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:22.039529 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:22.039790 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 01:17:22.040058 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":301,"threads_started":1}
I20260925 01:17:22.041688 26823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:22.123796 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 26700
I20260925 01:17:22.140098 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 26833
Sep 25 01:17:22 dist-test-slave-s12s krb5kdc[26459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:17:22.155784 26983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:22.141123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50856 (local address 127.20.102.131:35105)
0925 01:17:22.141865 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:22.141873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:22.141901 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:17:22.141962 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:22.141969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:22.142227 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:22.142364 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:22.143485 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:22.144510 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:22.145822 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:22.146500 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:22.146603 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:22.150427 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:22.150438 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:22.150452 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:22.150514 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:22.153422 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:22.154044 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:22.154051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:22.154053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:22.154178 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:22.154481 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:22.154484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:22.154485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:22.154866 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:22.155105 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:22.155309 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 01:17:22.155505 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":433,"threads_started":1}
I20260925 01:17:22.157053 26956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:22.238435 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 26833
I20260925 01:17:22.255076 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 26472
I20260925 01:17:22.256232 26534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:22.359174 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 26472
2026-09-25T01:17:22Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3870 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:17:22 dist-test-slave-s12s krb5kdc[26988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:17:22 dist-test-slave-s12s krb5kdc[26988](info): set up 2 sockets
Sep 25 01:17:22 dist-test-slave-s12s krb5kdc[26988](info): commencing operation
krb5kdc: starting...
W20260925 01:17:23.441416 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:17:23 dist-test-slave-s12s krb5kdc[26988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:17:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:17:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.190.keytab.
Entry for principal oryx/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.190.keytab.
I20260925 01:17:23.652156 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:37687
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.190.keytab
--principal=oryx/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:37687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:23.969313 27001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:23.969599 27001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:23.969749 27001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:23.976842 27001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:23.976940 27001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:23.977039 27001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:23.977128 27001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:23.977203 27001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:17:23.986778 27001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:37687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.190.keytab
--principal=oryx/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:37687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:23.988060 27001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:23.989320 27001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:24.000649 27006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:24.000624 27007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:24.000639 27009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:24.001626 27001 server_base.cc:1034] running on GCE node
I20260925 01:17:24.002380 27001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:24.003834 27001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:24.005064 27001 hybrid_clock.cc:648] HybridClock initialized: now 1790299044005012 us; error 54 us; skew 500 ppm
I20260925 01:17:24.005396 27001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:24 dist-test-slave-s12s krb5kdc[26988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299044, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:24.008405 27001 init.cc:377] Logged in from keytab as oryx/127.20.102.190@KRBTEST.COM (short username oryx)
I20260925 01:17:24.009924 27001 webserver.cc:460] Webserver started at http://127.20.102.190:33989/ using document root <none> and password file <none>
I20260925 01:17:24.010591 27001 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:24.010730 27001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:24.011034 27001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:24.014025 27001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "874aa79fff1748e58a20f66897829f27"
format_stamp: "Formatted at 2026-09-25 01:17:24 on dist-test-slave-s12s"
server_key: "d3ee6d25cc68e23b0db925c4761f65de"
server_key_iv: "9fe6fcc411958ecae9d6b4c8ca0728a4"
server_key_version: "encryptionkey@0"
I20260925 01:17:24.014765 27001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "874aa79fff1748e58a20f66897829f27"
format_stamp: "Formatted at 2026-09-25 01:17:24 on dist-test-slave-s12s"
server_key: "d3ee6d25cc68e23b0db925c4761f65de"
server_key_iv: "9fe6fcc411958ecae9d6b4c8ca0728a4"
server_key_version: "encryptionkey@0"
I20260925 01:17:24.019744 27001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:17:24.022940 27016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:24.024195 27001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 01:17:24.024374 27001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "874aa79fff1748e58a20f66897829f27"
format_stamp: "Formatted at 2026-09-25 01:17:24 on dist-test-slave-s12s"
server_key: "d3ee6d25cc68e23b0db925c4761f65de"
server_key_iv: "9fe6fcc411958ecae9d6b4c8ca0728a4"
server_key_version: "encryptionkey@0"
I20260925 01:17:24.024564 27001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:24.058939 27001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:24.062868 27001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:24.063160 27001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:24.081471 27001 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:37687
I20260925 01:17:24.081508 27067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:37687 every 8 connection(s)
I20260925 01:17:24.082847 27001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:17:24.086472 27068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:24.090209 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 27001
I20260925 01:17:24.090425 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:17:24.091220 20890 external_mini_cluster.cc:1428] Setting key f9c4470fe642c81127930fee5c354ff4
I20260925 01:17:24.099748 27068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27: Bootstrap starting.
I20260925 01:17:24.103526 27068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:24.104863 27068 log.cc:824] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:17:24 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
I20260925 01:17:24.107679 27068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27: No bootstrap required, opened a new log
I20260925 01:17:24.114027 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:24.093279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56016 (local address 127.20.102.190:37687)
0925 01:17:24.094009 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:24.094020 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:24.094061 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:17:24.095697 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:24.095728 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:24.096156 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:24.096476 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:24.098217 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:24.099474 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:24.101205 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:24.101860 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:24.101982 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:24.106808 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:24.106815 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:24.106828 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:24.106864 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:24.110041 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:24.110744 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:24.110750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:24.110756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:24.110916 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:24.111302 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:24.111306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:24.111308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:24.111855 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:24.112197 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:24.112651 (+ 454us) server_negotiation.cc:299] Negotiation successful
0925 01:17:24.113054 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":333,"threads_started":1}
I20260925 01:17:24.115499 27068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } }
I20260925 01:17:24.115872 27068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:24.115988 27068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 874aa79fff1748e58a20f66897829f27, State: Initialized, Role: FOLLOWER
I20260925 01:17:24.116528 27068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } }
I20260925 01:17:24.116812 27068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:17:24.116961 27068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:17:24.117117 27068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:24.119076 27068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } }
I20260925 01:17:24.119479 27068 leader_election.cc:302] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 874aa79fff1748e58a20f66897829f27; no voters:
I20260925 01:17:24.119990 27068 leader_election.cc:288] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:17:24.120146 27073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:24.120873 27073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [term 1 LEADER]: Becoming Leader. State: Replica: 874aa79fff1748e58a20f66897829f27, State: Running, Role: LEADER
I20260925 01:17:24.121464 27073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } }
I20260925 01:17:24.122160 27068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:24.124233 27074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } }
I20260925 01:17:24.124642 27074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:24.125798 27075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 874aa79fff1748e58a20f66897829f27. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } }
I20260925 01:17:24.126268 27075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:24.127370 27082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:17:24.131740 27082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:17:24.139923 27082 catalog_manager.cc:1347] Generated new cluster ID: 6a6d5cf20145481db80f59d94174d925
I20260925 01:17:24.140045 27082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:17:24.190591 27082 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:17:24.191946 27082 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:17:24.199575 27082 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27: Generated new TSK 0
I20260925 01:17:24.200368 27082 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.129.keytab.
Entry for principal oryx/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.129.keytab.
I20260925 01:17:24.264120 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37687
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.129.keytab
--principal=oryx/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:24.594981 27094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:24.595300 27094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:24.595506 27094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:24.602690 27094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:24.602810 27094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:24.602938 27094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:17:24.613010 27094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.129.keytab
--principal=oryx/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:24.614318 27094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:24.615818 27094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:24.629024 27102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:24.629142 27100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:24.629638 27099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:24.630172 27094 server_base.cc:1034] running on GCE node
I20260925 01:17:24.630753 27094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:24.631633 27094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:24.632843 27094 hybrid_clock.cc:648] HybridClock initialized: now 1790299044632800 us; error 52 us; skew 500 ppm
I20260925 01:17:24.633143 27094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:24 dist-test-slave-s12s krb5kdc[26988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299044, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:24.636236 27094 init.cc:377] Logged in from keytab as oryx/127.20.102.129@KRBTEST.COM (short username oryx)
I20260925 01:17:24.637658 27094 webserver.cc:460] Webserver started at http://127.20.102.129:36669/ using document root <none> and password file <none>
I20260925 01:17:24.638309 27094 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:24.638443 27094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:24.638772 27094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:24.641672 27094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "6804c301ed464eaaa95340cae0f1b656"
format_stamp: "Formatted at 2026-09-25 01:17:24 on dist-test-slave-s12s"
server_key: "e256919a25b1d10a56a94fce9ffc5785"
server_key_iv: "0067b9867059b96287c4985784b83d0f"
server_key_version: "encryptionkey@0"
I20260925 01:17:24.642360 27094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "6804c301ed464eaaa95340cae0f1b656"
format_stamp: "Formatted at 2026-09-25 01:17:24 on dist-test-slave-s12s"
server_key: "e256919a25b1d10a56a94fce9ffc5785"
server_key_iv: "0067b9867059b96287c4985784b83d0f"
server_key_version: "encryptionkey@0"
I20260925 01:17:24.647282 27094 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:17:24.650636 27109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:24.651890 27094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:17:24.652066 27094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "6804c301ed464eaaa95340cae0f1b656"
format_stamp: "Formatted at 2026-09-25 01:17:24 on dist-test-slave-s12s"
server_key: "e256919a25b1d10a56a94fce9ffc5785"
server_key_iv: "0067b9867059b96287c4985784b83d0f"
server_key_version: "encryptionkey@0"
I20260925 01:17:24.652254 27094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:24.672995 27094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:24.676949 27094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:24.677232 27094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:24.678205 27094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:24.679498 27094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:24.679584 27094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:24.679736 27094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:24.679800 27094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:24.712137 27094 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:38049
I20260925 01:17:24.712224 27222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:38049 every 8 connection(s)
I20260925 01:17:24.713567 27094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:17:24.715701 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 27094
I20260925 01:17:24.715906 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:17:24.716725 20890 external_mini_cluster.cc:1428] Setting key c87cbbb00f9bfb207c8365e4b5d67daf
Sep 25 01:17:24 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299044, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
I20260925 01:17:24.733567 27225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:24.716125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:47507 (local address 127.20.102.190:37687)
0925 01:17:24.716649 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:24.716655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:24.717762 (+ 1107us) server_negotiation.cc:407] Connection header received
0925 01:17:24.719570 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:24.719576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:24.719665 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:24.719776 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:24.721356 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:24.722167 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:24.723877 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:24.724299 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:24.724420 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:24.727930 (+ 3510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:24.727935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:24.727938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:24.727970 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:24.730620 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:24.731302 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:24.731307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:24.731310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:24.731426 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:24.731839 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:24.731844 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:24.731847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:24.732113 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:24.732266 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:24.732989 (+ 723us) server_negotiation.cc:299] Negotiation successful
0925 01:17:24.733210 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":228,"threads_started":1}
I20260925 01:17:24.734468 27226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:24.716797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.129:47507)
0925 01:17:24.717619 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:24.717666 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:24.719301 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:24.720022 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:24.720030 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:24.720532 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:24.721174 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:24.721197 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:24.722337 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:24.722341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:24.723730 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:24.723742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:24.724518 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:24.724527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:24.724812 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:24.725525 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:24.725555 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:24.727715 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:24.730749 (+ 3034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:24.730757 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:24.730772 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:24.731181 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:24.731531 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:24.731536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:24.731539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:24.731724 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:24.732230 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:24.732239 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:24.732575 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 01:17:24.732870 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 01:17:24.733377 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":327,"threads_started":1}
WARNING: no policy specified for oryx/127.20.102.130@KRBTEST.COM; defaulting to no policy
I20260925 01:17:24.736035 27223 heartbeater.cc:347] Connected to a master server at 127.20.102.190:37687
I20260925 01:17:24.736346 27223 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:24.737144 27223 heartbeater.cc:511] Master 127.20.102.190:37687 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.102.130@KRBTEST.COM" created.
I20260925 01:17:24.739478 27033 ts_manager.cc:194] Registered new tserver with Master: 6804c301ed464eaaa95340cae0f1b656 (127.20.102.129:38049)
I20260925 01:17:24.740917 27033 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:47507
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.130.keytab.
Entry for principal oryx/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.130.keytab.
I20260925 01:17:24.761332 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37687
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.130.keytab
--principal=oryx/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:25.093650 27229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:25.093950 27229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:25.094108 27229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:25.101466 27229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:25.101603 27229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:25.101781 27229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:17:25.111349 27229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.130.keytab
--principal=oryx/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:25.112600 27229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:25.113884 27229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:25.127116 27234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:25.127283 27237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:25.127911 27235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:25.128407 27229 server_base.cc:1034] running on GCE node
I20260925 01:17:25.129114 27229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:25.130039 27229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:25.131268 27229 hybrid_clock.cc:648] HybridClock initialized: now 1790299045131197 us; error 68 us; skew 500 ppm
I20260925 01:17:25.131593 27229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:25 dist-test-slave-s12s krb5kdc[26988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299045, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:25.134897 27229 init.cc:377] Logged in from keytab as oryx/127.20.102.130@KRBTEST.COM (short username oryx)
I20260925 01:17:25.136473 27229 webserver.cc:460] Webserver started at http://127.20.102.130:37509/ using document root <none> and password file <none>
I20260925 01:17:25.137214 27229 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:25.137356 27229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:25.137652 27229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:25.140616 27229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "2338a1bec88f4c1d89f7b19f487fab32"
format_stamp: "Formatted at 2026-09-25 01:17:25 on dist-test-slave-s12s"
server_key: "ae130bc721c34811314ce8bec0b5a0d5"
server_key_iv: "8b9470f68fa5e6ca9767d265a1f305c5"
server_key_version: "encryptionkey@0"
I20260925 01:17:25.141436 27229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "2338a1bec88f4c1d89f7b19f487fab32"
format_stamp: "Formatted at 2026-09-25 01:17:25 on dist-test-slave-s12s"
server_key: "ae130bc721c34811314ce8bec0b5a0d5"
server_key_iv: "8b9470f68fa5e6ca9767d265a1f305c5"
server_key_version: "encryptionkey@0"
I20260925 01:17:25.146584 27229 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:17:25.150058 27244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:25.151513 27229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:17:25.151695 27229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "2338a1bec88f4c1d89f7b19f487fab32"
format_stamp: "Formatted at 2026-09-25 01:17:25 on dist-test-slave-s12s"
server_key: "ae130bc721c34811314ce8bec0b5a0d5"
server_key_iv: "8b9470f68fa5e6ca9767d265a1f305c5"
server_key_version: "encryptionkey@0"
I20260925 01:17:25.151888 27229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:25.175040 27229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:25.179129 27229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:25.179416 27229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:25.180632 27229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:25.182093 27229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:25.182193 27229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:25.182349 27229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:25.182418 27229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:25.215758 27229 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:43921
I20260925 01:17:25.215797 27356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:43921 every 8 connection(s)
I20260925 01:17:25.217130 27229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:17:25.220815 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 27229
I20260925 01:17:25.221009 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:17:25.221494 20890 external_mini_cluster.cc:1428] Setting key 843921ed0be9623b1b66c294ea9f8aff
Sep 25 01:17:25 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299045, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
I20260925 01:17:25.241252 27225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:25.219848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:34741 (local address 127.20.102.190:37687)
0925 01:17:25.220032 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:25.220037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:25.221318 (+ 1281us) server_negotiation.cc:407] Connection header received
0925 01:17:25.223618 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:25.223625 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:25.223714 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:25.223823 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:25.225986 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:25.226684 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:25.229250 (+ 2566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:25.229761 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:25.229868 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:25.234084 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:25.234093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:25.234101 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:25.234149 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:25.237902 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:25.238601 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:25.238609 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:25.238615 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:25.238770 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:25.239196 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:25.239203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:25.239209 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:25.239584 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:25.239731 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:25.240483 (+ 752us) server_negotiation.cc:299] Negotiation successful
0925 01:17:25.240783 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:17:25.241921 27359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:25.220281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.130:34741)
0925 01:17:25.221145 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:25.221192 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:25.223349 (+ 2157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:25.224112 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:25.224124 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:25.224954 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:25.225820 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:25.225849 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:25.226847 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:25.226855 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:25.229108 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:25.229127 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:25.229982 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:25.229991 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:25.230321 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:25.231153 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:25.231185 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:25.233833 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:25.238034 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:25.238041 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:25.238054 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:25.238473 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:25.238882 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:25.238885 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:25.238887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:25.239062 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:25.239698 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:25.239703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:25.240113 (+ 410us) client_negotiation.cc:769] Sending connection context
0925 01:17:25.240341 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 01:17:25.240791 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":381,"threads_started":1}
I20260925 01:17:25.243386 27357 heartbeater.cc:347] Connected to a master server at 127.20.102.190:37687
I20260925 01:17:25.243749 27357 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:25.244753 27357 heartbeater.cc:511] Master 127.20.102.190:37687 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.20.102.131@KRBTEST.COM; defaulting to no policy
I20260925 01:17:25.246461 27033 ts_manager.cc:194] Registered new tserver with Master: 2338a1bec88f4c1d89f7b19f487fab32 (127.20.102.130:43921)
I20260925 01:17:25.247364 27033 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:34741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.131.keytab.
Entry for principal oryx/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.131.keytab.
I20260925 01:17:25.271814 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37687
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.131.keytab
--principal=oryx/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:25.598745 27362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:25.599084 27362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:25.599251 27362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:25.606334 27362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:25.606478 27362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:25.606652 27362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:17:25.616854 27362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.131.keytab
--principal=oryx/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37687
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:25.618175 27362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:25.619521 27362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:25.633055 27368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:25.633169 27367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:25.633049 27370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:25.634459 27362 server_base.cc:1034] running on GCE node
I20260925 01:17:25.635097 27362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:25.636126 27362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:25.637372 27362 hybrid_clock.cc:648] HybridClock initialized: now 1790299045637319 us; error 67 us; skew 500 ppm
I20260925 01:17:25.637698 27362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:25 dist-test-slave-s12s krb5kdc[26988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299045, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:25.640924 27362 init.cc:377] Logged in from keytab as oryx/127.20.102.131@KRBTEST.COM (short username oryx)
I20260925 01:17:25.642445 27362 webserver.cc:460] Webserver started at http://127.20.102.131:43099/ using document root <none> and password file <none>
I20260925 01:17:25.643159 27362 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:25.643301 27362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:25.643610 27362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:25.646664 27362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "03178d57cd3d4370aa02d2a3dbcadf1a"
format_stamp: "Formatted at 2026-09-25 01:17:25 on dist-test-slave-s12s"
server_key: "b1a4947afcf4f9f540ffc71637213959"
server_key_iv: "a36a65e95b668ae5ac7bcf4fd52fe48e"
server_key_version: "encryptionkey@0"
I20260925 01:17:25.647447 27362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "03178d57cd3d4370aa02d2a3dbcadf1a"
format_stamp: "Formatted at 2026-09-25 01:17:25 on dist-test-slave-s12s"
server_key: "b1a4947afcf4f9f540ffc71637213959"
server_key_iv: "a36a65e95b668ae5ac7bcf4fd52fe48e"
server_key_version: "encryptionkey@0"
I20260925 01:17:25.652653 27362 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:17:25.656119 27377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:25.657524 27362 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:25.657680 27362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "03178d57cd3d4370aa02d2a3dbcadf1a"
format_stamp: "Formatted at 2026-09-25 01:17:25 on dist-test-slave-s12s"
server_key: "b1a4947afcf4f9f540ffc71637213959"
server_key_iv: "a36a65e95b668ae5ac7bcf4fd52fe48e"
server_key_version: "encryptionkey@0"
I20260925 01:17:25.657868 27362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:25.682530 27362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:25.686841 27362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:25.687151 27362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:25.688109 27362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:25.689432 27362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:25.689509 27362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:25.689715 27362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:25.689769 27362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:25.722623 27362 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:36489
I20260925 01:17:25.722681 27489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:36489 every 8 connection(s)
I20260925 01:17:25.723846 27362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:17:25.733080 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 27362
I20260925 01:17:25.733270 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:17:25.733687 20890 external_mini_cluster.cc:1428] Setting key 9b8ebe50d6ded3df6ad5ed3c1d0b1373
Sep 25 01:17:25 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299045, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
I20260925 01:17:25.742970 27225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:25.726466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:49047 (local address 127.20.102.190:37687)
0925 01:17:25.726688 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:25.726694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:25.727912 (+ 1218us) server_negotiation.cc:407] Connection header received
0925 01:17:25.729833 (+ 1921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:25.729837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:25.729929 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:25.730026 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:25.731398 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:25.732125 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:25.733794 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:25.734338 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:25.734519 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:25.737752 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:25.737756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:25.737759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:25.737791 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:25.740324 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:25.741078 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:25.741085 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:25.741093 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:25.741245 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:25.741636 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:25.741639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:25.741641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:25.741897 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:25.742007 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:25.742576 (+ 569us) server_negotiation.cc:299] Negotiation successful
0925 01:17:25.742727 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:17:25.744185 27223 heartbeater.cc:503] Master 127.20.102.190:37687 was elected leader, sending a full tablet report...
I20260925 01:17:25.744396 27492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:25.726932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.131:49047)
0925 01:17:25.727721 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:25.727765 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:25.729584 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:25.730226 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:25.730233 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:25.730705 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:25.731252 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:25.731270 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:25.732244 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:25.732248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:25.733654 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:25.733666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:25.734562 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:25.734570 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:25.734818 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:25.735457 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:25.735479 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:25.737579 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:25.740451 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:25.740457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:25.740490 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:25.740931 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:25.741341 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:25.741345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:25.741347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:25.741540 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:25.742002 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:25.742008 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:25.742269 (+ 261us) client_negotiation.cc:769] Sending connection context
0925 01:17:25.742500 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 01:17:25.743028 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":273,"threads_started":1}
I20260925 01:17:25.746104 27490 heartbeater.cc:347] Connected to a master server at 127.20.102.190:37687
I20260925 01:17:25.746411 27490 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:25.747328 27490 heartbeater.cc:511] Master 127.20.102.190:37687 requested a full tablet report, sending...
I20260925 01:17:25.748914 27032 ts_manager.cc:194] Registered new tserver with Master: 03178d57cd3d4370aa02d2a3dbcadf1a (127.20.102.131:36489)
I20260925 01:17:25.749717 27032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:49047
I20260925 01:17:25.762379 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.102.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.102.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.102.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.189.keytab.
Entry for principal oryx/127.20.102.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.189.keytab.
I20260925 01:17:25.811868 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.189:35253
--webserver_interface=127.20.102.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.189.keytab
--principal=oryx/127.20.102.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:37687,127.20.102.189:35253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:26.139163 27495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:26.139515 27495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:26.139665 27495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:26.147217 27495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:26.147364 27495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:26.147464 27495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:26.147565 27495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:26.147642 27495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:17:26.157874 27495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:37687,127.20.102.189:35253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.189.keytab
--principal=oryx/127.20.102.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.189:35253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.102.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
W20260925 01:17:26.158285 27495 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.20.102.190:37687,127.20.102.189:35253'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 01:17:26.159354 27495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:26.160789 27495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:26.172253 27503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:26.172338 27501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:26.172791 27500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:26.172992 27495 server_base.cc:1034] running on GCE node
I20260925 01:17:26.173827 27495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:26.175195 27495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:26.176438 27495 hybrid_clock.cc:648] HybridClock initialized: now 1790299046176372 us; error 37 us; skew 500 ppm
I20260925 01:17:26.176868 27495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:26 dist-test-slave-s12s krb5kdc[26988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299046, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:26.180020 27495 init.cc:377] Logged in from keytab as oryx/127.20.102.189@KRBTEST.COM (short username oryx)
I20260925 01:17:26.181481 27495 webserver.cc:460] Webserver started at http://127.20.102.189:34647/ using document root <none> and password file <none>
I20260925 01:17:26.182122 27495 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:26.182206 27495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:26.182463 27495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:26.185370 27495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/data/instance:
uuid: "e99c1f0b8d2a428784076443be082d04"
format_stamp: "Formatted at 2026-09-25 01:17:26 on dist-test-slave-s12s"
server_key: "a93acd74436dd995eee4a0c70b88046d"
server_key_iv: "fa69b5fddb4e9d4ab32ba110ff2e7a01"
server_key_version: "encryptionkey@0"
I20260925 01:17:26.186084 27495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/wal/instance:
uuid: "e99c1f0b8d2a428784076443be082d04"
format_stamp: "Formatted at 2026-09-25 01:17:26 on dist-test-slave-s12s"
server_key: "a93acd74436dd995eee4a0c70b88046d"
server_key_iv: "fa69b5fddb4e9d4ab32ba110ff2e7a01"
server_key_version: "encryptionkey@0"
I20260925 01:17:26.191754 27495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:17:26.196197 27510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:26.197723 27495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:26.197901 27495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/wal
uuid: "e99c1f0b8d2a428784076443be082d04"
format_stamp: "Formatted at 2026-09-25 01:17:26 on dist-test-slave-s12s"
server_key: "a93acd74436dd995eee4a0c70b88046d"
server_key_iv: "fa69b5fddb4e9d4ab32ba110ff2e7a01"
server_key_version: "encryptionkey@0"
I20260925 01:17:26.198105 27495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:26.226008 27495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:26.230700 27495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:26.230917 27495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:26.248360 27495 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.189:35253
I20260925 01:17:26.248381 27561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.189:35253 every 8 connection(s)
I20260925 01:17:26.249778 27495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/data/info.pb
I20260925 01:17:26.250226 27357 heartbeater.cc:503] Master 127.20.102.190:37687 was elected leader, sending a full tablet report...
I20260925 01:17:26.250990 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 27495
I20260925 01:17:26.254575 27562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:26.261858 27562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:17:26 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.102.189@KRBTEST.COM
Sep 25 01:17:26 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299046, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.189@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
I20260925 01:17:26.275496 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.253838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45704 (local address 127.20.102.189:35253)
0925 01:17:26.254598 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:26.254610 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:26.254690 (+ 80us) server_negotiation.cc:407] Connection header received
0925 01:17:26.256507 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:26.256530 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:26.257015 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:26.257431 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:26.258939 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.260313 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.261699 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.262527 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.262718 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.267286 (+ 4568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:26.267294 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:26.267304 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:26.267342 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:26.270903 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.271745 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.271759 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.271771 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.272033 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.272418 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.272425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.272430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.273200 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:26.273437 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:26.273984 (+ 547us) server_negotiation.cc:299] Negotiation successful
0925 01:17:26.274413 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":341,"threads_started":1}
I20260925 01:17:26.280272 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.264300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56028)
0925 01:17:26.265071 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:26.265118 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:26.265251 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:26.265923 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:26.265930 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:26.266627 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:26.266864 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.266872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.267903 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.267907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.269802 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.269819 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.270556 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.270563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.270766 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.271422 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:26.271442 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:26.274326 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:26.277786 (+ 3460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:26.277795 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:26.277800 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:26.278422 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:26.278805 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:26.278808 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:26.278811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:26.278939 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:26.279468 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:26.279474 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:26.279638 (+ 164us) client_negotiation.cc:769] Sending connection context
0925 01:17:26.279790 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 01:17:26.280010 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":397,"threads_started":1}
I20260925 01:17:26.280289 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.264330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.20.102.190:37687)
0925 01:17:26.265620 (+ 1290us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:26.265624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:26.265639 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:17:26.265689 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:26.265692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:26.265747 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:26.265862 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:26.267013 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.267748 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.269957 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.270357 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.270453 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.274508 (+ 4055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:26.274513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:26.274517 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:26.274554 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:26.277641 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.278572 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.278575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.278578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.278710 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.279037 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.279040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.279042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.279324 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:26.279437 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:26.279889 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 01:17:26.280049 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1181,"thread_start_us":1093,"threads_started":1}
I20260925 01:17:26.281893 27562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:17:26 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299046, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.189@KRBTEST.COM for oryx/127.20.102.189@KRBTEST.COM
I20260925 01:17:26.293717 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.282781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:35253 (local address 127.0.0.1:45714)
0925 01:17:26.282947 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:26.282967 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:26.283101 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:26.283358 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:26.283361 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:26.283701 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:26.283822 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.283829 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.284792 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.284797 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.285635 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.285645 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.286291 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.286296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.286450 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.286917 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:26.286940 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:26.288960 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:26.291489 (+ 2529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:26.291494 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:26.291498 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:26.291856 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:26.292193 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:26.292199 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:26.292202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:26.292322 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:26.292833 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:26.292838 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:26.292965 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 01:17:26.293094 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 01:17:26.293354 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 01:17:26.293921 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.282764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.20.102.189:35253)
0925 01:17:26.282947 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:26.282953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:26.283033 (+ 80us) server_negotiation.cc:407] Connection header received
0925 01:17:26.283197 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:26.283207 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:26.283255 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:26.283344 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:26.283947 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.284622 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.285764 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.286153 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.286269 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.289123 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:26.289127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:26.289130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:26.289156 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:26.291352 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.291964 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.291972 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.291975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.292093 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.292415 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.292422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.292425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.292671 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:26.292856 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:26.293165 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 01:17:26.293496 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":54}
I20260925 01:17:26.298466 27562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: Bootstrap starting.
I20260925 01:17:26.301570 27562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:26.302635 27562 log.cc:824] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:26.305277 27562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: No bootstrap required, opened a new log
I20260925 01:17:26.311960 27562 raft_consensus.cc:348] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } }
I20260925 01:17:26.312294 27562 raft_consensus.cc:374] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:26.312395 27562 raft_consensus.cc:729] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e99c1f0b8d2a428784076443be082d04, State: Initialized, Role: FOLLOWER
I20260925 01:17:26.313030 27562 consensus_queue.cc:260] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } }
I20260925 01:17:26.313869 27569 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } } }
I20260925 01:17:26.314147 27569 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:17:26.314814 27562 sys_catalog.cc:564] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 01:17:26.323637 27580 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:17:26.323768 27580 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:17:26.325692 27495 master_runner.cc:428] Detected that this master e99c1f0b8d2a428784076443be082d04 is joining an existing cluster
I20260925 01:17:26.325811 27495 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.102.189:35253
I20260925 01:17:26.351505 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.330124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.20.102.189:35253)
0925 01:17:26.330300 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:26.330306 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:26.336914 (+ 6608us) server_negotiation.cc:407] Connection header received
0925 01:17:26.337412 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:26.337419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:26.337493 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:26.337555 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:26.339104 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.340336 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.342050 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.342651 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.342743 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.344762 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:26.344768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:26.344772 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:26.344807 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:26.348391 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.349266 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.349272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.349275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.349429 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.349851 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.349856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.349859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.350211 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:26.350373 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:26.350884 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 01:17:26.351098 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:17:26.352555 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.329710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:35253 (local address 127.0.0.1:45722)
0925 01:17:26.336819 (+ 7109us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:26.336841 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:26.337283 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:26.337810 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:26.337817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:26.338506 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:26.338946 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.338962 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.340554 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.340561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.341876 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.341892 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.342991 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.342999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.343189 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.343800 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:26.343819 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:26.344566 (+ 747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:26.348536 (+ 3970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:26.348553 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:26.348557 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:26.349118 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:26.349573 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:26.349577 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:26.349581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:26.349742 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:26.350448 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:26.350455 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:26.350624 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 01:17:26.351801 (+ 1177us) client_negotiation.cc:240] Negotiation successful
0925 01:17:26.352097 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6935,"thread_start_us":10120,"threads_started":1}
WARNING: no policy specified for oryx/127.20.102.188@KRBTEST.COM; defaulting to no policy
I20260925 01:17:26.357841 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.329579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56038 (local address 127.20.102.190:37687)
0925 01:17:26.329759 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:26.329768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:26.339816 (+ 10048us) server_negotiation.cc:407] Connection header received
0925 01:17:26.340243 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:26.340250 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:26.340330 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:26.340403 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:26.341914 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.342957 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.345874 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.346936 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.347055 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.349679 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:26.349687 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:26.349694 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:26.349738 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:26.354196 (+ 4458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.355392 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.355398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.355402 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.355568 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.356077 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.356082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.356084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.356462 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:26.356610 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:26.357274 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 01:17:26.357508 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:17:26.357936 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.330158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56038)
0925 01:17:26.339715 (+ 9557us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:26.339732 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:26.340082 (+ 350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:26.340618 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:26.340622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:26.341374 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:26.341717 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.341732 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.343121 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.343129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.345433 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.345718 (+ 285us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.347373 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.347382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.347598 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.348231 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:26.348258 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:26.349469 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:26.354691 (+ 5222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:26.354697 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:26.354702 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:26.355238 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:26.355732 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:26.355737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:26.355742 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:26.355913 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:26.356718 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:26.356725 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:26.356961 (+ 236us) client_negotiation.cc:769] Sending connection context
0925 01:17:26.357128 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 01:17:26.357420 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9428,"thread_start_us":6756,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.102.188@KRBTEST.COM" created.
Sep 25 01:17:26 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299044, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.190@KRBTEST.COM for oryx/127.20.102.189@KRBTEST.COM
I20260925 01:17:26.384182 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.364181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.20.102.189:35253)
0925 01:17:26.364346 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:26.364351 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:26.365390 (+ 1039us) server_negotiation.cc:407] Connection header received
0925 01:17:26.365763 (+ 373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:26.365774 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:26.365865 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:26.365941 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:26.368477 (+ 2536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.369797 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.371685 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.372495 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.372622 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.377150 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:26.377157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:26.377162 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:26.377201 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:26.381092 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.381933 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.381939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.381942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.382089 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.382524 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.382533 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.382535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.382848 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:26.383003 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:26.383576 (+ 573us) server_negotiation.cc:299] Negotiation successful
0925 01:17:26.383808 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:17:26.384241 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.364129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:35253 (local address 127.0.0.1:45730)
0925 01:17:26.365256 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:26.365300 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:26.365528 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:26.366212 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:26.366220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:26.367992 (+ 1772us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:26.368289 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.368304 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.369936 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.369942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.371526 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.371552 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.372660 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.372666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.372871 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.373590 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:26.373614 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:26.376916 (+ 3302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:26.381230 (+ 4314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:26.381243 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:26.381249 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:26.381809 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:26.382205 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:26.382211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:26.382215 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:26.382411 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:26.382978 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:26.382986 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:26.383234 (+ 248us) client_negotiation.cc:769] Sending connection context
0925 01:17:26.383469 (+ 235us) client_negotiation.cc:240] Negotiation successful
0925 01:17:26.383804 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"thread_start_us":725,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.102.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.188.keytab.
Entry for principal oryx/127.20.102.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.188.keytab.
I20260925 01:17:26.386490 27032 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.102.189:35253: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e99c1f0b8d2a428784076443be082d04"
member_type: NON_VOTER
last_known_addr {
host: "127.20.102.189"
port: 35253
}
attrs {
promote: true
}
}
dest_uuid: "874aa79fff1748e58a20f66897829f27"
cas_config_opid_index: -1
I20260925 01:17:26.387565 27032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: NON_VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: true } }
I20260925 01:17:26.390280 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.188:39747
--webserver_interface=127.20.102.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.188.keytab
--principal=oryx/127.20.102.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:37687,127.20.102.189:35253,127.20.102.188:39747
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:17:26.400354 27569 raft_consensus.cc:482] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:17:26.400799 27569 raft_consensus.cc:504] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } }
I20260925 01:17:26.402809 27569 leader_election.cc:288] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 874aa79fff1748e58a20f66897829f27 (127.20.102.190:37687)
I20260925 01:17:26.403852 27043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e99c1f0b8d2a428784076443be082d04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "874aa79fff1748e58a20f66897829f27" is_pre_election: true
I20260925 01:17:26.405265 27514 leader_election.cc:302] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: e99c1f0b8d2a428784076443be082d04; no voters: 874aa79fff1748e58a20f66897829f27
I20260925 01:17:26.406139 27569 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 01:17:26.408468 27537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:26.408191 27593 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e99c1f0b8d2a428784076443be082d04 (127.20.102.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: NON_VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: true } } }
I20260925 01:17:26.412592 27593 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.102.189:35253
I20260925 01:17:26.412772 27537 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 FOLLOWER]: Refusing update from remote peer 874aa79fff1748e58a20f66897829f27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 01:17:26.413445 27495 master.cc:537] Master@127.20.102.189:35253 shutting down...
I20260925 01:17:26.414062 27593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: NON_VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: true } } }
I20260925 01:17:26.414397 27593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:26.417275 27595 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: NON_VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:17:26.423951 27019 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 01:17:26.424978 27495 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:26.425467 27495 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:26.425715 27495 tablet_replica.cc:330] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: stopping tablet replica
W20260925 01:17:26.428894 27019 consensus_peers.cc:487] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 -> Peer e99c1f0b8d2a428784076443be082d04 (127.20.102.189:35253): Couldn't send request to peer e99c1f0b8d2a428784076443be082d04. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 01:17:26.433476 27495 master.cc:552] Master@127.20.102.189:35253 shutdown complete.
I20260925 01:17:26.433632 27495 master_runner.cc:305] Clearing existing system tablet
I20260925 01:17:26.434154 27495 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:26.434320 27495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:26.437909 27495 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 01:17:26.439841 27599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:26.440178 27495 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:26.440616 27495 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 01:17:26.440775 27495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/wal
uuid: "e99c1f0b8d2a428784076443be082d04"
format_stamp: "Formatted at 2026-09-25 01:17:26 on dist-test-slave-s12s"
server_key: "a93acd74436dd995eee4a0c70b88046d"
server_key_iv: "fa69b5fddb4e9d4ab32ba110ff2e7a01"
server_key_version: "encryptionkey@0"
I20260925 01:17:26.446054 27495 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:26.452927 27495 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 01:17:26.453169 27495 log.cc:1196] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 01:17:26.453646 27495 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: Deleting consensus metadata
I20260925 01:17:26.453948 27495 master_runner.cc:315] Copying system tablet from 127.20.102.190:37687
I20260925 01:17:26.456775 27495 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: tablet copy: Beginning tablet copy session from remote peer at address 127.20.102.190:37687
I20260925 01:17:26.476258 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.457664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56044 (local address 127.20.102.190:37687)
0925 01:17:26.457844 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:26.457850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:26.461249 (+ 3399us) server_negotiation.cc:407] Connection header received
0925 01:17:26.461505 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:26.461513 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:26.461614 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:26.461750 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:26.462772 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.464067 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.466184 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.466854 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.467008 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.469428 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:26.469435 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:26.469440 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:26.469487 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:26.472923 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.473786 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.473791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.473795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.473940 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.474460 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.474465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.474468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.474811 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:26.474958 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:26.475643 (+ 685us) server_negotiation.cc:299] Negotiation successful
0925 01:17:26.475874 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:17:26.476995 27605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.457638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56044)
0925 01:17:26.461084 (+ 3446us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:26.461109 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:26.461265 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:26.461779 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:26.461784 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:26.462345 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:26.462608 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.462624 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.464243 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.464249 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.465994 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.466012 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.467049 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.467055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.467251 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.467981 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:26.468028 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:26.469119 (+ 1091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:26.473074 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:26.473081 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:26.473087 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:26.473644 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:26.474106 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:26.474112 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:26.474117 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:26.474316 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:26.474948 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:26.474956 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:26.475210 (+ 254us) client_negotiation.cc:769] Sending connection context
0925 01:17:26.475469 (+ 259us) client_negotiation.cc:240] Negotiation successful
0925 01:17:26.476044 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3305,"thread_start_us":3208,"threads_started":1}
I20260925 01:17:26.478230 27053 tablet_copy_service.cc:140] P 874aa79fff1748e58a20f66897829f27: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e99c1f0b8d2a428784076443be082d04 ({username='oryx', principal='oryx/127.20.102.189@KRBTEST.COM'} at 127.0.0.1:56044)
I20260925 01:17:26.478510 27053 tablet_copy_service.cc:161] P 874aa79fff1748e58a20f66897829f27: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e99c1f0b8d2a428784076443be082d04 at {username='oryx', principal='oryx/127.20.102.189@KRBTEST.COM'} at 127.0.0.1:56044: session id = e99c1f0b8d2a428784076443be082d04-00000000000000000000000000000000
I20260925 01:17:26.483083 27053 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 01:17:26.485394 27495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 01:17:26.492194 27495 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: tablet copy: Starting download of 0 data blocks...
I20260925 01:17:26.492575 27495 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: tablet copy: Starting download of 1 WAL segments...
I20260925 01:17:26.495393 27495 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 01:17:26.499879 27053 tablet_copy_service.cc:342] P 874aa79fff1748e58a20f66897829f27: Request end of tablet copy session e99c1f0b8d2a428784076443be082d04-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.102.189@KRBTEST.COM'} at 127.0.0.1:56044
I20260925 01:17:26.500052 27053 tablet_copy_service.cc:433] P 874aa79fff1748e58a20f66897829f27: ending tablet copy session e99c1f0b8d2a428784076443be082d04-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e99c1f0b8d2a428784076443be082d04
I20260925 01:17:26.512627 27495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:26.515286 27606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:26.516546 27607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:26.520834 27609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:26.523344 27495 server_base.cc:1034] running on GCE node
I20260925 01:17:26.523775 27495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:26.524547 27495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:26.525811 27495 hybrid_clock.cc:648] HybridClock initialized: now 1790299046525736 us; error 44 us; skew 500 ppm
I20260925 01:17:26.526113 27495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:26 dist-test-slave-s12s krb5kdc[26988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299046, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:26.528476 27495 init.cc:377] Logged in from keytab as oryx/127.20.102.189@KRBTEST.COM (short username oryx)
I20260925 01:17:26.533706 27495 webserver.cc:460] Webserver started at http://127.20.102.189:39835/ using document root <none> and password file <none>
I20260925 01:17:26.534195 27495 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:26.534343 27495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:26.537473 27495 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 01:17:26.539515 27616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:26.539978 27495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260925 01:17:26.540189 27495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/wal
uuid: "e99c1f0b8d2a428784076443be082d04"
format_stamp: "Formatted at 2026-09-25 01:17:26 on dist-test-slave-s12s"
server_key: "a93acd74436dd995eee4a0c70b88046d"
server_key_iv: "fa69b5fddb4e9d4ab32ba110ff2e7a01"
server_key_version: "encryptionkey@0"
I20260925 01:17:26.540483 27495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:26.577144 27495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:26.582190 27495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:26.617674 27495 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.189:35253
I20260925 01:17:26.617731 27667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.189:35253 every 8 connection(s)
I20260925 01:17:26.619015 27495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-1/data/info.pb
I20260925 01:17:26.624781 27668 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:17:26.629392 27668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: Bootstrap starting.
I20260925 01:17:26.645498 27668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:17:26.645983 27668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: Bootstrap complete.
I20260925 01:17:26.646826 27668 raft_consensus.cc:348] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: NON_VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: true } }
I20260925 01:17:26.647074 27668 raft_consensus.cc:729] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e99c1f0b8d2a428784076443be082d04, State: Initialized, Role: LEARNER
I20260925 01:17:26.647300 27668 consensus_queue.cc:260] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: NON_VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: true } }
I20260925 01:17:26.648113 27671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: NON_VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: true } } }
I20260925 01:17:26.648471 27668 sys_catalog.cc:564] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:26.648636 27671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: This master's current role is: LEARNER
I20260925 01:17:26.657325 27682 catalog_manager.cc:1259] Loaded cluster ID: 6a6d5cf20145481db80f59d94174d925
I20260925 01:17:26.657423 27682 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: loading cluster ID for follower catalog manager: success
I20260925 01:17:26.659864 27682 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: acquiring CA information for follower catalog manager: success
I20260925 01:17:26.661803 27682 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:17:26.752059 27490 heartbeater.cc:503] Master 127.20.102.190:37687 was elected leader, sending a full tablet report...
I20260925 01:17:26.862494 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.855502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:35253 (local address 127.0.0.1:45732)
0925 01:17:26.855806 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:26.855832 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:26.856052 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:26.857175 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:26.857180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:26.857218 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:26.857417 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.857426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.858773 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.858778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.860005 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.860017 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.861566 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.861572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.861847 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.861932 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 01:17:26.862018 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 01:17:26.862193 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260925 01:17:26.863044 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.855606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45732 (local address 127.20.102.189:35253)
0925 01:17:26.856850 (+ 1244us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:26.856854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:26.856871 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:17:26.856930 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:26.856934 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:26.857001 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:26.857137 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:26.857584 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.858621 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.860159 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.861307 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.862247 (+ 940us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.862413 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:26.862537 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 01:17:26.862683 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1142,"thread_start_us":965,"threads_started":1}
I20260925 01:17:26.864013 27643 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 01:17:26.864596 27671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 874aa79fff1748e58a20f66897829f27. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: NON_VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: true } } }
I20260925 01:17:26.864885 27671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: This master's current role is: LEARNER
W20260925 01:17:26.871240 27594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:26.871495 27594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:26.871599 27594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:26.878790 27594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:26.878881 27594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:26.878926 27594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:26.878983 27594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:26.879027 27594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:17:26.888628 27594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:37687,127.20.102.189:35253,127.20.102.188:39747
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.188.keytab
--principal=oryx/127.20.102.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.188:39747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.102.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:26.889882 27594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:26.891122 27594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:26.902741 27688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:26.902678 27689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:26.902768 27691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:26.903462 27594 server_base.cc:1034] running on GCE node
I20260925 01:17:26.904273 27594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:26.906155 27594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:26.907401 27594 hybrid_clock.cc:648] HybridClock initialized: now 1790299046907350 us; error 46 us; skew 500 ppm
I20260925 01:17:26.907773 27594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:26 dist-test-slave-s12s krb5kdc[26988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299046, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:26.910956 27594 init.cc:377] Logged in from keytab as oryx/127.20.102.188@KRBTEST.COM (short username oryx)
I20260925 01:17:26.912417 27594 webserver.cc:460] Webserver started at http://127.20.102.188:34295/ using document root <none> and password file <none>
I20260925 01:17:26.913147 27594 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:26.913285 27594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:26.913582 27594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:26.916512 27594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/data/instance:
uuid: "4281250057b94e258de2aac45dd6870f"
format_stamp: "Formatted at 2026-09-25 01:17:26 on dist-test-slave-s12s"
server_key: "26e992e5a0b47d28c390631786ef6986"
server_key_iv: "bf64a5fa30a1fb6d5f4286a2486aaf13"
server_key_version: "encryptionkey@0"
I20260925 01:17:26.917364 27594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/wal/instance:
uuid: "4281250057b94e258de2aac45dd6870f"
format_stamp: "Formatted at 2026-09-25 01:17:26 on dist-test-slave-s12s"
server_key: "26e992e5a0b47d28c390631786ef6986"
server_key_iv: "bf64a5fa30a1fb6d5f4286a2486aaf13"
server_key_version: "encryptionkey@0"
I20260925 01:17:26.922792 27594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:17:26.926288 27698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:26.927594 27594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:17:26.927734 27594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/wal
uuid: "4281250057b94e258de2aac45dd6870f"
format_stamp: "Formatted at 2026-09-25 01:17:26 on dist-test-slave-s12s"
server_key: "26e992e5a0b47d28c390631786ef6986"
server_key_iv: "bf64a5fa30a1fb6d5f4286a2486aaf13"
server_key_version: "encryptionkey@0"
I20260925 01:17:26.927946 27594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:26.942559 27594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:26.948446 27594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:26.948832 27594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:26.967407 27594 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.188:39747
I20260925 01:17:26.967453 27749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.188:39747 every 8 connection(s)
I20260925 01:17:26.969003 27594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/data/info.pb
I20260925 01:17:26.972815 27750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:26.978597 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 27594
I20260925 01:17:26.979432 27750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:17:26 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299046, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.188@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
Sep 25 01:17:26 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.102.188@KRBTEST.COM
I20260925 01:17:27.000356 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.982462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.20.102.190:37687)
0925 01:17:26.983873 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:26.983879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:26.983900 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:17:26.984166 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:26.984172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:26.984252 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:26.984372 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:26.986170 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.987338 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.989446 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.990022 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.990153 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.994141 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:26.994146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:26.994149 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:26.994181 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:26.997021 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.997867 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.997871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.997873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.997983 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.998461 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.998466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.998469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.998846 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:26.998996 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:26.999797 (+ 801us) server_negotiation.cc:299] Negotiation successful
0925 01:17:26.999952 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1290,"thread_start_us":1189,"threads_started":1}
I20260925 01:17:27.001475 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.982331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56050)
0925 01:17:26.983167 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:26.983221 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:26.983932 (+ 711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:26.984463 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:26.984469 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:26.985429 (+ 960us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:26.985986 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.986017 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.987483 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.987487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.989307 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:26.989321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.990228 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.990232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:26.990393 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.991160 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:26.991192 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:26.993945 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:26.997147 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:26.997157 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:26.997175 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:26.997752 (+ 577us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:26.998093 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:26.998098 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:26.998103 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:26.998337 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:26.998971 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:26.998976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:26.999401 (+ 425us) client_negotiation.cc:769] Sending connection context
0925 01:17:26.999695 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 01:17:27.000193 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":379,"threads_started":1}
I20260925 01:17:27.001863 27753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:26.980874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.20.102.188:39747)
0925 01:17:26.981551 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:26.981561 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:26.981596 (+ 35us) server_negotiation.cc:407] Connection header received
0925 01:17:26.983926 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:26.983956 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:26.984232 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:26.984426 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:26.986269 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.987802 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.989596 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:26.990395 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:26.990484 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:26.995293 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:26.995301 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:26.995312 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:26.995340 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:26.998337 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.999099 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.999106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.999109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.999255 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:26.999597 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:26.999603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:26.999606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:26.999988 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:27.000271 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:27.001129 (+ 858us) server_negotiation.cc:299] Negotiation successful
0925 01:17:27.001383 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":293,"threads_started":1}
I20260925 01:17:27.003190 27750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:17:27 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299046, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.188@KRBTEST.COM for oryx/127.20.102.189@KRBTEST.COM
I20260925 01:17:27.015376 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.003958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:35253 (local address 127.0.0.1:45748)
0925 01:17:27.004292 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:27.004324 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:27.004436 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:27.004763 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:27.004771 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:27.005079 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:27.005198 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.005205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.006116 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.006119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.007007 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.007017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.007760 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.007764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.007889 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.008391 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:27.008407 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:27.010457 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:27.013241 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.013245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.013248 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.013638 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.014054 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.014057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.014059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.014174 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.014721 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:27.014725 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:27.014837 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 01:17:27.014946 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 01:17:27.015141 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260925 01:17:27.015579 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.003947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.20.102.189:35253)
0925 01:17:27.004153 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:27.004161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:27.004412 (+ 251us) server_negotiation.cc:407] Connection header received
0925 01:17:27.004532 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:27.004536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:27.004616 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:27.004751 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:27.005311 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.006000 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.007184 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.007635 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.007746 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.010649 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:27.010654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:27.010658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:27.010690 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:27.013118 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.013774 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.013779 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.013782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.013943 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.014272 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.014276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.014279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.014601 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:27.014740 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:27.015075 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 01:17:27.015274 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:17:27.016489 27750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:17:27 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299046, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.188@KRBTEST.COM for oryx/127.20.102.188@KRBTEST.COM
I20260925 01:17:27.027971 27754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.017177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:39747 (local address 127.0.0.1:41744)
0925 01:17:27.017323 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:27.017341 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:27.017436 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:27.017670 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:27.017673 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:27.017968 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:27.018097 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.018104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.018968 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.018971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.019799 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.019809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.020455 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.020459 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.020637 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.021212 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:27.021229 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:27.023368 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:27.026090 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.026093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.026096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.026416 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.026712 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.026715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.026717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.026858 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.027290 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:27.027293 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:27.027402 (+ 109us) client_negotiation.cc:769] Sending connection context
0925 01:17:27.027533 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 01:17:27.027700 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 01:17:27.028326 27753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.017183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.20.102.188:39747)
0925 01:17:27.017335 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:27.017339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:27.017401 (+ 62us) server_negotiation.cc:407] Connection header received
0925 01:17:27.017516 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:27.017519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:27.017568 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:27.017643 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:27.018207 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.018864 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.019934 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.020334 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.020435 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.023534 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:27.023538 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:27.023542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:27.023573 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:27.025972 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.026509 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.026513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.026515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.026625 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.026942 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.026945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.026947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.027200 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:27.027303 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:27.027603 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 01:17:27.027951 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":47}
I20260925 01:17:27.033164 27750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: Bootstrap starting.
I20260925 01:17:27.036335 27750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:27.037343 27750 log.cc:824] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:27.039789 27750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: No bootstrap required, opened a new log
I20260925 01:17:27.046918 27750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } }
I20260925 01:17:27.047214 27750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:27.047298 27750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4281250057b94e258de2aac45dd6870f, State: Initialized, Role: FOLLOWER
I20260925 01:17:27.047840 27750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } }
I20260925 01:17:27.048647 27757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } } }
I20260925 01:17:27.049082 27757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:17:27.049739 27750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:27.056715 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 27094
W20260925 01:17:27.059618 27768 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:17:27.059769 27768 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:17:27.063812 27594 master_runner.cc:428] Detected that this master 4281250057b94e258de2aac45dd6870f is joining an existing cluster
I20260925 01:17:27.063922 27594 master_runner.cc:432] Initiating AddMaster RPC to add 127.20.102.188:39747
Sep 25 01:17:27 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.102.129@KRBTEST.COM
I20260925 01:17:27.080833 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.057865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51310 (local address 127.20.102.129:38049)
0925 01:17:27.060384 (+ 2519us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:27.060390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:27.060414 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:17:27.060468 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:27.060474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:27.060749 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:27.061076 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:27.062716 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.064295 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.066210 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.067212 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.067345 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.073081 (+ 5736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:27.073096 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:27.073115 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:27.073162 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:27.077192 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.078098 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.078107 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.078112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.078269 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.078758 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.078764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.078767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.079199 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:27.079495 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:27.080110 (+ 615us) server_negotiation.cc:299] Negotiation successful
0925 01:17:27.080383 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2361,"thread_start_us":2201,"threads_started":1}
I20260925 01:17:27.082338 27217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:27.082895 27757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:17:27.083288 27757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } }
I20260925 01:17:27.085206 27757 leader_election.cc:288] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 874aa79fff1748e58a20f66897829f27 (127.20.102.190:37687), e99c1f0b8d2a428784076443be082d04 (127.20.102.189:35253)
I20260925 01:17:27.086182 27643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4281250057b94e258de2aac45dd6870f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e99c1f0b8d2a428784076443be082d04" is_pre_election: true
I20260925 01:17:27.087527 27043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4281250057b94e258de2aac45dd6870f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "874aa79fff1748e58a20f66897829f27" is_pre_election: true
I20260925 01:17:27.088990 27702 leader_election.cc:302] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4281250057b94e258de2aac45dd6870f; no voters: 874aa79fff1748e58a20f66897829f27, e99c1f0b8d2a428784076443be082d04
I20260925 01:17:27.089854 27757 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 01:17:27.100750 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.068918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56056 (local address 127.20.102.190:37687)
0925 01:17:27.069121 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:27.069126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:27.073372 (+ 4246us) server_negotiation.cc:407] Connection header received
0925 01:17:27.075356 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:27.075361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:27.075439 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:27.075507 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:27.080807 (+ 5300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.081907 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.084342 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.085040 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.085138 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.092764 (+ 7626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:27.092771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:27.092776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:27.092818 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:27.096845 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.098436 (+ 1591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.098442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.098446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.098604 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.099057 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.099062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.099066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.099418 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:27.099571 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:27.100123 (+ 552us) server_negotiation.cc:299] Negotiation successful
0925 01:17:27.100342 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 01:17:27.100801 27683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.069033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45760 (local address 127.20.102.189:35253)
0925 01:17:27.069201 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:27.069209 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:27.074888 (+ 5679us) server_negotiation.cc:407] Connection header received
0925 01:17:27.075157 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:27.075163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:27.075280 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:27.075349 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:27.077756 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.079106 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.081116 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.081717 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.081814 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.092149 (+ 10335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:27.092159 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:27.092167 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:27.092221 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:27.096650 (+ 4429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.097729 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.097738 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.097744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.097931 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.098519 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.098528 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.098534 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.099002 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:27.099169 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:27.099919 (+ 750us) server_negotiation.cc:299] Negotiation successful
0925 01:17:27.100254 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:17:27.102305 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.069531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:35253 (local address 127.0.0.1:45760)
0925 01:17:27.074769 (+ 5238us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:27.074792 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:27.075009 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:27.075577 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:27.075582 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:27.076472 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:27.077545 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.077561 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.079305 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.079313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.080930 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.080945 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.082117 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.082123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.082305 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.090446 (+ 8141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:27.090479 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:27.091870 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:27.096868 (+ 4998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.096877 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.096884 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.097554 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.098124 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.098131 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.098138 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.098370 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.099273 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:27.099281 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:27.099527 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 01:17:27.101102 (+ 1575us) client_negotiation.cc:240] Negotiation successful
0925 01:17:27.101565 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5068,"mutex_wait_us":644,"thread_start_us":3348,"threads_started":1}
I20260925 01:17:27.102306 27776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.069524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56056)
0925 01:17:27.073272 (+ 3748us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:27.073290 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:27.075097 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:27.075736 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:27.075743 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:27.076648 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:27.077143 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.077159 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.082351 (+ 5192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.082356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.084186 (+ 1830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.084201 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.085476 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.085483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.085671 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.086251 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:27.086274 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:27.092491 (+ 6217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:27.097866 (+ 5375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.097871 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.097875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.098307 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.098760 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.098765 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.098769 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.098935 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.099653 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:27.099657 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:27.099855 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 01:17:27.101324 (+ 1469us) client_negotiation.cc:240] Negotiation successful
0925 01:17:27.101837 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3588,"mutex_wait_us":114,"thread_start_us":9806,"threads_started":1}
I20260925 01:17:27.120553 27753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.073731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.20.102.188:39747)
0925 01:17:27.073941 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:27.073949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:27.079218 (+ 5269us) server_negotiation.cc:407] Connection header received
0925 01:17:27.079378 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:27.079385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:27.079464 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:27.079580 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:27.105895 (+ 26315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.107382 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.109954 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.110973 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.111104 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.113811 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:27.113820 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:27.113828 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:27.113865 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:27.117583 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.118371 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.118376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.118380 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.118531 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.118906 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.118911 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.118914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.119262 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:27.119425 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:27.119922 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 01:17:27.120144 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 01:17:27.121536 27775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.073826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:39747 (local address 127.0.0.1:41756)
0925 01:17:27.079129 (+ 5303us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:27.079151 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:27.079269 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:27.104352 (+ 25083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:27.104361 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:27.105279 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:27.105695 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.105720 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.107657 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.107666 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.109780 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.109798 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.111480 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.111486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.111706 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.112414 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:27.112462 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:27.113636 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:27.117728 (+ 4092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.117734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.117739 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.118252 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.118663 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.118667 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.118671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.118802 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.119500 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:27.119505 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:27.119654 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 01:17:27.120746 (+ 1092us) client_negotiation.cc:240] Negotiation successful
0925 01:17:27.121038 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5177,"thread_start_us":566,"threads_started":1}
Sep 25 01:17:27 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299044, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.190@KRBTEST.COM for oryx/127.20.102.188@KRBTEST.COM
I20260925 01:17:27.148880 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.108445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41770 (local address 127.20.102.188:39747)
0925 01:17:27.112889 (+ 4444us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:27.112897 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:27.112952 (+ 55us) server_negotiation.cc:407] Connection header received
0925 01:17:27.113311 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:27.113360 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:27.113710 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:27.113933 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:27.133160 (+ 19227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.134808 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.136696 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.137442 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.137743 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.141780 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:27.141787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:27.141792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:27.141856 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:27.145693 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.146557 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.146563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.146567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.146724 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.147170 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.147175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.147179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.147533 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:27.147674 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:27.148147 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 01:17:27.148364 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3494,"thread_start_us":3329,"threads_started":1}
I20260925 01:17:27.148910 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.107746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:39747 (local address 127.0.0.1:41770)
0925 01:17:27.108102 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:27.108122 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:27.108232 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:27.113967 (+ 5735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:27.113972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:27.114460 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:27.114718 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.114731 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.135007 (+ 20276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.135015 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.136479 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.136498 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.137738 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.137745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.137922 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.138656 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:27.138680 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:27.141527 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:27.145865 (+ 4338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.145871 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.145875 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.146400 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.146860 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.146865 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.146869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.147054 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.147683 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:27.147689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:27.147869 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 01:17:27.148034 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 01:17:27.148465 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260925 01:17:27.150870 27032 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.20.102.188:39747: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4281250057b94e258de2aac45dd6870f"
member_type: NON_VOTER
last_known_addr {
host: "127.20.102.188"
port: 39747
}
attrs {
promote: true
}
}
dest_uuid: "874aa79fff1748e58a20f66897829f27"
cas_config_opid_index: 5
I20260925 01:17:27.151957 27032 consensus_queue.cc:237] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: NON_VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: true } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } }
I20260925 01:17:27.153733 27595 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4281250057b94e258de2aac45dd6870f (127.20.102.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: NON_VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: true } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } } }
I20260925 01:17:27.155423 27643 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 LEARNER]: Refusing update from remote peer 874aa79fff1748e58a20f66897829f27: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 01:17:27.157096 27725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:27.164398 27725 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 1 FOLLOWER]: Refusing update from remote peer 874aa79fff1748e58a20f66897829f27: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 01:17:27.166167 27780 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: NON_VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 01:17:27.172976 27595 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.20.102.188:39747
I20260925 01:17:27.173807 27594 master.cc:537] Master@127.20.102.188:39747 shutting down...
I20260925 01:17:27.173723 27595 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:27.177306 27781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: NON_VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: true } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } } }
I20260925 01:17:27.177706 27781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:27.178121 27643 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4281250057b94e258de2aac45dd6870f (127.20.102.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: NON_VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: true } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } } }
I20260925 01:17:27.182374 27671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: NON_VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: true } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } } }
I20260925 01:17:27.182673 27671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: This master's current role is: LEARNER
I20260925 01:17:27.183213 27780 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27: attempting to promote NON_VOTER e99c1f0b8d2a428784076443be082d04 to VOTER
I20260925 01:17:27.185571 27594 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:27.185930 27594 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:27.186144 27594 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: stopping tablet replica
I20260925 01:17:27.186129 27780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } }
I20260925 01:17:27.188221 27643 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 LEARNER]: Refusing update from remote peer 874aa79fff1748e58a20f66897829f27: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 01:17:27.188781 27595 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 01:17:27.191015 27595 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e99c1f0b8d2a428784076443be082d04 (127.20.102.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } } }
I20260925 01:17:27.194939 27781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } } }
I20260925 01:17:27.195207 27781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:27.195358 27595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } } }
I20260925 01:17:27.195604 27595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:27.196483 27642 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, e99c1f0b8d2a428784076443be082d04 (127.20.102.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } } }
I20260925 01:17:27.200309 27671 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } } }
I20260925 01:17:27.200551 27671 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 01:17:27.202178 27017 consensus_peers.cc:487] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 -> Peer 4281250057b94e258de2aac45dd6870f (127.20.102.188:39747): Couldn't send request to peer 4281250057b94e258de2aac45dd6870f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 01:17:27.215400 27594 master.cc:552] Master@127.20.102.188:39747 shutdown complete.
I20260925 01:17:27.215500 27594 master_runner.cc:305] Clearing existing system tablet
I20260925 01:17:27.215899 27594 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:27.216001 27594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:27.219919 27594 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:17:27.221451 27787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:27.221890 27594 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:27.222153 27594 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260925 01:17:27.222296 27594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/wal
uuid: "4281250057b94e258de2aac45dd6870f"
format_stamp: "Formatted at 2026-09-25 01:17:26 on dist-test-slave-s12s"
server_key: "26e992e5a0b47d28c390631786ef6986"
server_key_iv: "bf64a5fa30a1fb6d5f4286a2486aaf13"
server_key_version: "encryptionkey@0"
I20260925 01:17:27.226251 27594 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:27.231659 27594 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 01:17:27.231832 27594 log.cc:1196] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 01:17:27.232282 27594 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: Deleting consensus metadata
I20260925 01:17:27.232520 27594 master_runner.cc:315] Copying system tablet from 127.20.102.190:37687
I20260925 01:17:27.235090 27594 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: tablet copy: Beginning tablet copy session from remote peer at address 127.20.102.190:37687
I20260925 01:17:27.250715 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 27094
I20260925 01:17:27.255085 27755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.236398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.20.102.190:37687)
0925 01:17:27.236744 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:27.236754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:27.241386 (+ 4632us) server_negotiation.cc:407] Connection header received
0925 01:17:27.241667 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:27.241673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:27.241749 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:27.241857 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:27.242947 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.243720 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.245931 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.246377 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.246526 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.248389 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:27.248399 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:27.248402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:27.248474 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:27.251853 (+ 3379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.252634 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.252639 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.252642 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.252891 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.253390 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.253398 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.253406 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.253763 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:27.253890 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:27.254480 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 01:17:27.254714 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260925 01:17:27.255555 27793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.235915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56058)
0925 01:17:27.241270 (+ 5355us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:27.241291 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:27.241464 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:27.241968 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:27.241976 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:27.242537 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:27.242783 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.242792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.243934 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.243940 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.245767 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.245780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.246543 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.246550 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.246745 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.247401 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:27.247418 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:27.248198 (+ 780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:27.251983 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.251991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.251996 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.252488 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.253066 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.253072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.253076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.253258 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.253996 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:27.254007 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:27.254203 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 01:17:27.254376 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 01:17:27.254724 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5197,"thread_start_us":5044,"threads_started":1}
I20260925 01:17:27.256085 27053 tablet_copy_service.cc:140] P 874aa79fff1748e58a20f66897829f27: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4281250057b94e258de2aac45dd6870f ({username='oryx', principal='oryx/127.20.102.188@KRBTEST.COM'} at 127.0.0.1:56058)
I20260925 01:17:27.256264 27053 tablet_copy_service.cc:161] P 874aa79fff1748e58a20f66897829f27: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4281250057b94e258de2aac45dd6870f at {username='oryx', principal='oryx/127.20.102.188@KRBTEST.COM'} at 127.0.0.1:56058: session id = 4281250057b94e258de2aac45dd6870f-00000000000000000000000000000000
I20260925 01:17:27.258648 27053 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 01:17:27.260203 27594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 01:17:27.265437 27594 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: tablet copy: Starting download of 0 data blocks...
I20260925 01:17:27.265724 27594 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: tablet copy: Starting download of 1 WAL segments...
I20260925 01:17:27.267843 27594 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 01:17:27.271112 27053 tablet_copy_service.cc:342] P 874aa79fff1748e58a20f66897829f27: Request end of tablet copy session 4281250057b94e258de2aac45dd6870f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.20.102.188@KRBTEST.COM'} at 127.0.0.1:56058
I20260925 01:17:27.271256 27053 tablet_copy_service.cc:433] P 874aa79fff1748e58a20f66897829f27: ending tablet copy session 4281250057b94e258de2aac45dd6870f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4281250057b94e258de2aac45dd6870f
I20260925 01:17:27.278558 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:38049
--local_ip_for_outbound_sockets=127.20.102.129
--tserver_master_addrs=127.20.102.190:37687
--webserver_port=36669
--webserver_interface=127.20.102.129
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.129.keytab
--principal=oryx/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.102.190:37687,127.20.102.189:35253
--tserver_master_addrs=127.20.102.190:37687,127.20.102.189:35253,127.20.102.188:39747 with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:17:27.287086 27594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:27.289702 27795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:27.298600 27796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:27.299065 27798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:27.299381 27594 server_base.cc:1034] running on GCE node
I20260925 01:17:27.299870 27594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:27.300511 27594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:27.301697 27594 hybrid_clock.cc:648] HybridClock initialized: now 1790299047301620 us; error 72 us; skew 500 ppm
I20260925 01:17:27.301936 27594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:27 dist-test-slave-s12s krb5kdc[26988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299047, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:27.303844 27594 init.cc:377] Logged in from keytab as oryx/127.20.102.188@KRBTEST.COM (short username oryx)
I20260925 01:17:27.309716 27594 webserver.cc:460] Webserver started at http://127.20.102.188:32915/ using document root <none> and password file <none>
I20260925 01:17:27.310058 27594 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:27.310190 27594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:27.313777 27594 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260925 01:17:27.315729 27805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:27.316221 27594 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 01:17:27.316363 27594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/wal
uuid: "4281250057b94e258de2aac45dd6870f"
format_stamp: "Formatted at 2026-09-25 01:17:26 on dist-test-slave-s12s"
server_key: "26e992e5a0b47d28c390631786ef6986"
server_key_iv: "bf64a5fa30a1fb6d5f4286a2486aaf13"
server_key_version: "encryptionkey@0"
I20260925 01:17:27.316565 27594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:27.327881 27594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:27.332073 27594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:27.376114 27594 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.188:39747
I20260925 01:17:27.376150 27856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.188:39747 every 8 connection(s)
I20260925 01:17:27.377151 27594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/master-2/data/info.pb
I20260925 01:17:27.384277 27857 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:17:27.389642 27857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: Bootstrap starting.
I20260925 01:17:27.404338 27857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:17:27.404876 27857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: Bootstrap complete.
I20260925 01:17:27.405755 27857 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } }
I20260925 01:17:27.406011 27857 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4281250057b94e258de2aac45dd6870f, State: Initialized, Role: LEARNER
I20260925 01:17:27.406248 27857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } }
I20260925 01:17:27.407078 27860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } } }
I20260925 01:17:27.407378 27857 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:27.407519 27860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [sys.catalog]: This master's current role is: LEARNER
I20260925 01:17:27.425117 27871 catalog_manager.cc:1259] Loaded cluster ID: 6a6d5cf20145481db80f59d94174d925
I20260925 01:17:27.425247 27871 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: loading cluster ID for follower catalog manager: success
I20260925 01:17:27.427807 27871 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: acquiring CA information for follower catalog manager: success
I20260925 01:17:27.429767 27871 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:17:27.743484 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.736771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41772 (local address 127.20.102.188:39747)
0925 01:17:27.737517 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:27.737521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:27.737539 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:17:27.737613 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:27.737616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:27.737704 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:27.737811 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:27.738295 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.739274 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.741551 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.742567 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.742759 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.742975 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:27.743122 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 01:17:27.743218 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":479,"threads_started":1}
I20260925 01:17:27.743518 27876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.736429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:39747 (local address 127.0.0.1:41772)
0925 01:17:27.736967 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:27.736993 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:27.737160 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:27.737866 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:27.737871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:27.737899 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:27.738142 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.738156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.739414 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.739421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.741381 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.741398 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.742773 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.742779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.742960 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.742995 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:17:27.743098 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 01:17:27.743232 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":274,"threads_started":1}
I20260925 01:17:27.744534 27832 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 01:17:27.745123 27860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 874aa79fff1748e58a20f66897829f27. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } } }
I20260925 01:17:27.745455 27860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [sys.catalog]: This master's current role is: LEARNER
W20260925 01:17:27.746924 27794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:27.747196 27794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:27.747363 27794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:27.755108 27794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:27.755270 27794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:27.755501 27794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:17:27.765440 27794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.129.keytab
--principal=oryx/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:38049
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=36669
--tserver_master_addrs=127.20.102.190:37687,127.20.102.189:35253,127.20.102.188:39747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:27.766762 27794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:27.768121 27794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:27.783185 27879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:27.784164 27880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:27.787360 27794 server_base.cc:1034] running on GCE node
W20260925 01:17:27.787777 27883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:27.788560 27794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:27.789682 27794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:27.790885 27794 hybrid_clock.cc:648] HybridClock initialized: now 1790299047790841 us; error 36 us; skew 500 ppm
I20260925 01:17:27.791215 27794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:27 dist-test-slave-s12s krb5kdc[26988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299047, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:27.794552 27794 init.cc:377] Logged in from keytab as oryx/127.20.102.129@KRBTEST.COM (short username oryx)
I20260925 01:17:27.796002 27794 webserver.cc:460] Webserver started at http://127.20.102.129:36669/ using document root <none> and password file <none>
I20260925 01:17:27.796806 27794 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:27.796952 27794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:27.802302 27794 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:17:27.805172 27890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:27.806526 27794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:27.806704 27794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "6804c301ed464eaaa95340cae0f1b656"
format_stamp: "Formatted at 2026-09-25 01:17:24 on dist-test-slave-s12s"
server_key: "e256919a25b1d10a56a94fce9ffc5785"
server_key_iv: "0067b9867059b96287c4985784b83d0f"
server_key_version: "encryptionkey@0"
I20260925 01:17:27.807793 27794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:27.827420 27794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:27.832146 27794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:27.832439 27794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:27.833473 27794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:27.835084 27794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:27.835223 27794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:27.835351 27794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:27.835438 27794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:27.871519 27794 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:38049
I20260925 01:17:27.871613 28002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:38049 every 8 connection(s)
I20260925 01:17:27.873065 27794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:17:27.875473 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 27794
I20260925 01:17:27.875840 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 27229
Sep 25 01:17:27 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299047, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.188@KRBTEST.COM
Sep 25 01:17:27 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299047, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.189@KRBTEST.COM
Sep 25 01:17:27 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299047, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
Sep 25 01:17:27 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.102.130@KRBTEST.COM
I20260925 01:17:27.916074 27877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.889753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:49329 (local address 127.20.102.188:39747)
0925 01:17:27.889980 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:27.889989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:27.890017 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:17:27.894555 (+ 4538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:27.894565 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:27.894680 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:27.894770 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:27.897606 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.898781 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.901627 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.902296 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.902395 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.908668 (+ 6273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:27.908690 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:27.908695 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:27.908738 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:27.912661 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.913667 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.913675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.913679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.913830 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.914330 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.914335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.914338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.914714 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:27.914860 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:27.915462 (+ 602us) server_negotiation.cc:299] Negotiation successful
0925 01:17:27.915701 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:17:27.917258 28013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.889222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:39747 (local address 127.20.102.129:49329)
0925 01:17:27.889882 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:27.889903 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:27.894135 (+ 4232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:27.895417 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:27.895432 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:27.896430 (+ 998us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:27.897398 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.897426 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.899008 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.899014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.901459 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.901474 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.902718 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.902725 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.903035 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.903825 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:27.903869 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:27.908402 (+ 4533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:27.912837 (+ 4435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.912842 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.912847 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.913527 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.913985 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.913991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.913995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.914147 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.914947 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:27.914952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:27.915117 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 01:17:27.916350 (+ 1233us) client_negotiation.cc:240] Negotiation successful
0925 01:17:27.916644 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":90,"thread_start_us":810,"threads_started":1}
I20260925 01:17:27.917671 28008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.876845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:55899 (local address 127.20.102.189:35253)
0925 01:17:27.897283 (+ 20438us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:27.897288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:27.897311 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:17:27.897368 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:27.897374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:27.897462 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:27.898175 (+ 713us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:27.899015 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.900240 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.902465 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.903121 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.903239 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.907937 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:27.907944 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:27.907948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:27.907989 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:27.911907 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.912893 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.912900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.912907 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.913064 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.913626 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.913632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.913635 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.914004 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:27.914152 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:27.916836 (+ 2684us) server_negotiation.cc:299] Negotiation successful
0925 01:17:27.917081 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20315,"thread_start_us":20186,"threads_started":1}
I20260925 01:17:27.917829 28010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.877500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:45943 (local address 127.20.102.190:37687)
0925 01:17:27.895796 (+ 18296us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:27.895803 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:27.895824 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:17:27.897245 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:27.897251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:27.897348 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:27.897456 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:27.898690 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.900240 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.902465 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.903121 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.903239 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.908865 (+ 5626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:27.908872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:27.908876 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:27.908920 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:27.913250 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.914085 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.914090 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.914093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.914263 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.914716 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.914723 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.914726 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.915054 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:27.915206 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:27.916863 (+ 1657us) server_negotiation.cc:299] Negotiation successful
0925 01:17:27.917371 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18156,"thread_start_us":19879,"threads_started":1}
I20260925 01:17:27.918210 28011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.877532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.129:45943)
0925 01:17:27.886932 (+ 9400us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:27.887011 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:27.896753 (+ 9742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:27.897638 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:27.897642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:27.898190 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:27.898478 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.898490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.900412 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.900417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.902292 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.902307 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.903522 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.903530 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.904826 (+ 1296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.905402 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:27.905433 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:27.908586 (+ 3153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:27.913397 (+ 4811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.913401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.913404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.913943 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.914386 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.914390 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.914393 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.914580 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.915266 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:27.915271 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:27.915418 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 01:17:27.915561 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 01:17:27.915808 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8997,"mutex_wait_us":1188,"thread_start_us":8716,"threads_started":1}
I20260925 01:17:27.918763 28012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.878171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:35253 (local address 127.20.102.129:55899)
0925 01:17:27.889175 (+ 11004us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:27.889192 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:27.894146 (+ 4954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:27.897634 (+ 3488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:27.897638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:27.898513 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:27.898851 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.898862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.900422 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.900426 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.902294 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:27.902307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.903549 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.903560 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:27.903788 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.904345 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:27.904364 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:27.907718 (+ 3354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:27.912087 (+ 4369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.912096 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.912111 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.912745 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.913202 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:27.913207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:27.913211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:27.913486 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:27.914235 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:27.914243 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:27.914653 (+ 410us) client_negotiation.cc:769] Sending connection context
0925 01:17:27.914976 (+ 323us) client_negotiation.cc:240] Negotiation successful
0925 01:17:27.915524 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10858,"mutex_wait_us":63,"thread_start_us":10602,"threads_started":1}
I20260925 01:17:27.919359 28009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:27.877004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.20.102.130:43921)
0925 01:17:27.890689 (+ 13685us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:27.890702 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:27.896633 (+ 5931us) server_negotiation.cc:407] Connection header received
0925 01:17:27.896745 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:27.896753 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:27.896996 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:27.897158 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:27.899565 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.901087 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.903035 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:27.903929 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:27.904069 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:27.910798 (+ 6729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:27.910813 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:27.910832 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:27.910886 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:27.915292 (+ 4406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.916170 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.916181 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.916184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.916341 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:27.916840 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:27.916845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:27.916849 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:27.917259 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:27.917590 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:27.918721 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0925 01:17:27.918968 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13484,"thread_start_us":13334,"threads_started":1}
I20260925 01:17:27.919348 28004 heartbeater.cc:347] Connected to a master server at 127.20.102.190:37687
I20260925 01:17:27.919409 28003 heartbeater.cc:347] Connected to a master server at 127.20.102.188:39747
I20260925 01:17:27.919688 28003 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:27.919693 28004 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:27.920509 28005 heartbeater.cc:347] Connected to a master server at 127.20.102.189:35253
I20260925 01:17:27.920650 28005 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:27.920732 28004 heartbeater.cc:511] Master 127.20.102.190:37687 requested a full tablet report, sending...
I20260925 01:17:27.920797 27352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:27.920836 28003 heartbeater.cc:511] Master 127.20.102.188:39747 requested a full tablet report, sending...
I20260925 01:17:27.921167 28005 heartbeater.cc:511] Master 127.20.102.189:35253 requested a full tablet report, sending...
I20260925 01:17:27.922869 27032 ts_manager.cc:194] Re-registered known tserver with Master: 6804c301ed464eaaa95340cae0f1b656 (127.20.102.129:38049)
I20260925 01:17:27.923228 27822 ts_manager.cc:194] Registered new tserver with Master: 6804c301ed464eaaa95340cae0f1b656 (127.20.102.129:38049)
I20260925 01:17:27.923260 27633 ts_manager.cc:194] Registered new tserver with Master: 6804c301ed464eaaa95340cae0f1b656 (127.20.102.129:38049)
I20260925 01:17:27.924266 27032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:45943
I20260925 01:17:28.010109 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 27229
I20260925 01:17:28.028844 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:43921
--local_ip_for_outbound_sockets=127.20.102.130
--tserver_master_addrs=127.20.102.190:37687
--webserver_port=37509
--webserver_interface=127.20.102.130
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.130.keytab
--principal=oryx/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.102.190:37687,127.20.102.189:35253
--tserver_master_addrs=127.20.102.190:37687,127.20.102.189:35253,127.20.102.188:39747 with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:17:28.264292 27872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: NON_VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: true } } }
I20260925 01:17:28.264717 27872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:28.266002 27872 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27: attempting to promote NON_VOTER 4281250057b94e258de2aac45dd6870f to VOTER
I20260925 01:17:28.266837 27872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } }
I20260925 01:17:28.269543 27642 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 FOLLOWER]: Refusing update from remote peer 874aa79fff1748e58a20f66897829f27: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 01:17:28.269553 27832 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 1 LEARNER]: Refusing update from remote peer 874aa79fff1748e58a20f66897829f27: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 01:17:28.270213 28017 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 01:17:28.270704 27872 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 01:17:28.274291 28017 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4281250057b94e258de2aac45dd6870f (127.20.102.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } } }
I20260925 01:17:28.274904 27642 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4281250057b94e258de2aac45dd6870f (127.20.102.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } } }
I20260925 01:17:28.278836 28017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } } }
I20260925 01:17:28.279155 28017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:28.279451 28017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } } }
I20260925 01:17:28.279713 28017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:28.285213 28022 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } } }
I20260925 01:17:28.285697 28022 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:17:28.286119 28017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } } }
I20260925 01:17:28.286396 28017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:28.286722 27832 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4281250057b94e258de2aac45dd6870f (127.20.102.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } } }
I20260925 01:17:28.290110 28023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "874aa79fff1748e58a20f66897829f27" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } } }
I20260925 01:17:28.290482 28023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [sys.catalog]: This master's current role is: FOLLOWER
W20260925 01:17:28.365993 28015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:28.366277 28015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:28.366375 28015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:28.373626 28015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:28.373725 28015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:28.373847 28015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:17:28.383679 28015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.130.keytab
--principal=oryx/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:43921
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=37509
--tserver_master_addrs=127.20.102.190:37687,127.20.102.189:35253,127.20.102.188:39747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:28.385105 28015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:28.386435 28015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:28.399969 28028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:28.400095 28031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:28.399907 28029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:28.401078 28015 server_base.cc:1034] running on GCE node
I20260925 01:17:28.401687 28015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:28.402602 28015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:28.403828 28015 hybrid_clock.cc:648] HybridClock initialized: now 1790299048403780 us; error 42 us; skew 500 ppm
I20260925 01:17:28.404222 28015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:28 dist-test-slave-s12s krb5kdc[26988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299048, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:28.407488 28015 init.cc:377] Logged in from keytab as oryx/127.20.102.130@KRBTEST.COM (short username oryx)
I20260925 01:17:28.409087 28015 webserver.cc:460] Webserver started at http://127.20.102.130:37509/ using document root <none> and password file <none>
I20260925 01:17:28.409770 28015 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:28.409911 28015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:28.415573 28015 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:17:28.418535 28038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:28.420044 28015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260925 01:17:28.420214 28015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "2338a1bec88f4c1d89f7b19f487fab32"
format_stamp: "Formatted at 2026-09-25 01:17:25 on dist-test-slave-s12s"
server_key: "ae130bc721c34811314ce8bec0b5a0d5"
server_key_iv: "8b9470f68fa5e6ca9767d265a1f305c5"
server_key_version: "encryptionkey@0"
I20260925 01:17:28.421339 28015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:28.446369 28015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:28.450555 28015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:28.450839 28015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:28.451920 28015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:28.453521 28015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:28.453662 28015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:28.453791 28015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:28.453886 28015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:28.490038 28015 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:43921
I20260925 01:17:28.490097 28150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:43921 every 8 connection(s)
I20260925 01:17:28.491605 28015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:17:28.500907 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 28015
I20260925 01:17:28.501284 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 27362
Sep 25 01:17:28 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299048, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.189@KRBTEST.COM
Sep 25 01:17:28 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299048, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
Sep 25 01:17:28 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299048, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.188@KRBTEST.COM
Sep 25 01:17:28 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.102.131@KRBTEST.COM
I20260925 01:17:28.530037 28158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:28.496310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:57121 (local address 127.20.102.190:37687)
0925 01:17:28.502242 (+ 5932us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:28.502252 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:28.506135 (+ 3883us) server_negotiation.cc:407] Connection header received
0925 01:17:28.506722 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:28.506728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:28.506834 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:28.506945 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:28.510355 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:28.511142 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:28.514487 (+ 3345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:28.515267 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:28.515371 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:28.521208 (+ 5837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:28.521215 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:28.521220 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:28.521273 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:28.526281 (+ 5008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:28.527092 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:28.527097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:28.527101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:28.527296 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:28.527806 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:28.527812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:28.527815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:28.528216 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:28.528347 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:28.529425 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0925 01:17:28.529652 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5749,"thread_start_us":5644,"threads_started":1}
I20260925 01:17:28.530058 28155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:28.495054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:56937 (local address 127.20.102.189:35253)
0925 01:17:28.503144 (+ 8090us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:28.503150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:28.504778 (+ 1628us) server_negotiation.cc:407] Connection header received
0925 01:17:28.506713 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:28.506722 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:28.506812 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:28.506897 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:28.511425 (+ 4528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:28.512651 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:28.514981 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:28.515681 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:28.515793 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:28.521750 (+ 5957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:28.521756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:28.521760 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:28.521799 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:28.525529 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:28.526731 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:28.526737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:28.526740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:28.527102 (+ 362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:28.527688 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:28.527694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:28.527699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:28.528086 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:28.528250 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:28.529433 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0925 01:17:28.529661 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7946,"thread_start_us":7814,"threads_started":1}
I20260925 01:17:28.530062 28156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:28.495463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:41775 (local address 127.20.102.188:39747)
0925 01:17:28.502704 (+ 7241us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:28.502714 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:28.505492 (+ 2778us) server_negotiation.cc:407] Connection header received
0925 01:17:28.506796 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:28.506802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:28.506891 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:28.506995 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:28.510335 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:28.511558 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:28.514523 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:28.515161 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:28.515298 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:28.521765 (+ 6467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:28.521771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:28.521775 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:28.521812 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:28.525421 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:28.526731 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:28.526737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:28.526740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:28.526893 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:28.527678 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:28.527686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:28.527690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:28.528078 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:28.528214 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:28.529418 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0925 01:17:28.529662 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7078,"thread_start_us":6931,"threads_started":1}
I20260925 01:17:28.531245 28162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:28.503081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.20.102.131:36489)
0925 01:17:28.506409 (+ 3328us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:28.506419 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:28.506461 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:17:28.506542 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:28.506553 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:28.506822 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:28.507762 (+ 940us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:28.509649 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:28.511415 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:28.513572 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:28.514440 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:28.514552 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:28.522415 (+ 7863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:28.522431 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:28.522448 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:28.522503 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:28.526867 (+ 4364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:28.527801 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:28.527810 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:28.527814 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:28.527970 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:28.528443 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:28.528447 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:28.528450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:28.528953 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:28.529313 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:28.530564 (+ 1251us) server_negotiation.cc:299] Negotiation successful
0925 01:17:28.530820 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3149,"thread_start_us":2982,"threads_started":1}
I20260925 01:17:28.532392 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:28.495642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:35253 (local address 127.20.102.130:56937)
0925 01:17:28.503536 (+ 7894us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:28.503603 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:28.506355 (+ 2752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:28.507489 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:28.507501 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:28.508707 (+ 1206us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:28.511248 (+ 2541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:28.511264 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:28.512894 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:28.512899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:28.514804 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:28.514819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:28.516132 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:28.516141 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:28.516355 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:28.517268 (+ 913us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:28.517292 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:28.521535 (+ 4243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:28.525696 (+ 4161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:28.525699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:28.525702 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:28.526584 (+ 882us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:28.527232 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:28.527237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:28.527241 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:28.527536 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:28.528367 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:28.528371 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:28.528913 (+ 542us) client_negotiation.cc:769] Sending connection context
0925 01:17:28.530394 (+ 1481us) client_negotiation.cc:240] Negotiation successful
0925 01:17:28.531001 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":2041,"spinlock_wait_cycles":94976,"thread_start_us":647,"threads_started":1}
I20260925 01:17:28.532637 27485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:28.532411 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:28.495642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.130:57121)
0925 01:17:28.506012 (+ 10370us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:28.506042 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:28.506366 (+ 324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:28.507490 (+ 1124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:28.507502 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:28.508710 (+ 1208us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:28.510111 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:28.510140 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:28.511307 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:28.511312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:28.514298 (+ 2986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:28.514314 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:28.515750 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:28.515758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:28.516153 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:28.517041 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:28.517072 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:28.520884 (+ 3812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:28.526466 (+ 5582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:28.526470 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:28.526474 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:28.526986 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:28.527467 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:28.527474 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:28.527478 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:28.527675 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:28.528397 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:28.528401 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:28.528913 (+ 512us) client_negotiation.cc:769] Sending connection context
0925 01:17:28.530263 (+ 1350us) client_negotiation.cc:240] Negotiation successful
0925 01:17:28.530893 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10200,"mutex_wait_us":1510,"spinlock_wait_cycles":33408,"thread_start_us":8163,"threads_started":1}
I20260925 01:17:28.533936 28152 heartbeater.cc:347] Connected to a master server at 127.20.102.190:37687
I20260925 01:17:28.534241 28152 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:28.535053 28152 heartbeater.cc:511] Master 127.20.102.190:37687 requested a full tablet report, sending...
I20260925 01:17:28.536008 28153 heartbeater.cc:347] Connected to a master server at 127.20.102.189:35253
I20260925 01:17:28.536142 28153 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:28.536540 28153 heartbeater.cc:511] Master 127.20.102.189:35253 requested a full tablet report, sending...
I20260925 01:17:28.536844 28160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:28.496426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:39747 (local address 127.20.102.130:41775)
0925 01:17:28.505291 (+ 8865us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:28.505310 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:28.506355 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:28.507489 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:28.507501 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:28.508725 (+ 1224us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:28.510112 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:28.510140 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:28.511746 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:28.511752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:28.514336 (+ 2584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:28.514353 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:28.515748 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:28.515757 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:28.516164 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:28.517150 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:28.517178 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:28.521534 (+ 4356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:28.525645 (+ 4111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:28.525656 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:28.525682 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:28.526584 (+ 902us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:28.527270 (+ 686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:28.527275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:28.527279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:28.527514 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:28.528351 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:28.528361 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:28.528923 (+ 562us) client_negotiation.cc:769] Sending connection context
0925 01:17:28.531014 (+ 2091us) client_negotiation.cc:240] Negotiation successful
0925 01:17:28.531514 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8697,"mutex_wait_us":473,"spinlock_wait_cycles":86016,"thread_start_us":9124,"threads_started":1}
I20260925 01:17:28.536871 27032 ts_manager.cc:194] Re-registered known tserver with Master: 2338a1bec88f4c1d89f7b19f487fab32 (127.20.102.130:43921)
I20260925 01:17:28.537688 28151 heartbeater.cc:347] Connected to a master server at 127.20.102.188:39747
I20260925 01:17:28.537822 28151 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:28.538012 27032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:57121
I20260925 01:17:28.538192 28151 heartbeater.cc:511] Master 127.20.102.188:39747 requested a full tablet report, sending...
I20260925 01:17:28.538843 27633 ts_manager.cc:194] Registered new tserver with Master: 2338a1bec88f4c1d89f7b19f487fab32 (127.20.102.130:43921)
I20260925 01:17:28.539269 27822 ts_manager.cc:194] Registered new tserver with Master: 2338a1bec88f4c1d89f7b19f487fab32 (127.20.102.130:43921)
I20260925 01:17:28.619302 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 27362
I20260925 01:17:28.636984 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:36489
--local_ip_for_outbound_sockets=127.20.102.131
--tserver_master_addrs=127.20.102.190:37687
--webserver_port=43099
--webserver_interface=127.20.102.131
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.131.keytab
--principal=oryx/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.20.102.190:37687,127.20.102.189:35253
--tserver_master_addrs=127.20.102.190:37687,127.20.102.189:35253,127.20.102.188:39747 with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:17:28.927606 28004 heartbeater.cc:503] Master 127.20.102.190:37687 was elected leader, sending a full tablet report...
W20260925 01:17:28.973987 28164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:28.974257 28164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:28.974350 28164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:28.981804 28164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:28.981907 28164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:28.982025 28164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:17:28.991459 28164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.131.keytab
--principal=oryx/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:36489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=43099
--tserver_master_addrs=127.20.102.190:37687,127.20.102.189:35253,127.20.102.188:39747
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:28.992779 28164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:28.994036 28164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:29.007097 28169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:29.007064 28172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:29.007143 28170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:29.008401 28164 server_base.cc:1034] running on GCE node
I20260925 01:17:29.009076 28164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:29.009944 28164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:29.011147 28164 hybrid_clock.cc:648] HybridClock initialized: now 1790299049011112 us; error 29 us; skew 500 ppm
I20260925 01:17:29.011467 28164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:29 dist-test-slave-s12s krb5kdc[26988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299049, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:29.014669 28164 init.cc:377] Logged in from keytab as oryx/127.20.102.131@KRBTEST.COM (short username oryx)
I20260925 01:17:29.016093 28164 webserver.cc:460] Webserver started at http://127.20.102.131:43099/ using document root <none> and password file <none>
I20260925 01:17:29.016862 28164 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:29.017032 28164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:29.022085 28164 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 01:17:29.024875 28179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:29.026237 28164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:17:29.026413 28164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "03178d57cd3d4370aa02d2a3dbcadf1a"
format_stamp: "Formatted at 2026-09-25 01:17:25 on dist-test-slave-s12s"
server_key: "b1a4947afcf4f9f540ffc71637213959"
server_key_iv: "a36a65e95b668ae5ac7bcf4fd52fe48e"
server_key_version: "encryptionkey@0"
I20260925 01:17:29.027383 28164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:29.046865 28164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:29.051360 28164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:29.051654 28164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:29.052646 28164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:29.054059 28164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:29.054174 28164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:29.054337 28164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:29.054422 28164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:29.087600 28164 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:36489
I20260925 01:17:29.087663 28291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:36489 every 8 connection(s)
I20260925 01:17:29.089205 28164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:17:29.097914 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 28164
Sep 25 01:17:29 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299049, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.189@KRBTEST.COM
Sep 25 01:17:29 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299049, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.188@KRBTEST.COM
Sep 25 01:17:29 dist-test-slave-s12s krb5kdc[26988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299049, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
I20260925 01:17:29.147945 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.093246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:54909 (local address 127.20.102.189:35253)
0925 01:17:29.111907 (+ 18661us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.111917 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.123819 (+ 11902us) server_negotiation.cc:407] Connection header received
0925 01:17:29.126706 (+ 2887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.126711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.126797 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.126904 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:29.129903 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.130979 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.133408 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.134053 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.134166 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.139536 (+ 5370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:29.139545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:29.139549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:29.139596 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:29.143365 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.144517 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.144524 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.144527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.144736 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.145271 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.145276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.145280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.145652 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:29.145805 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.147302 (+ 1497us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.147538 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18492,"thread_start_us":18675,"threads_started":1}
I20260925 01:17:29.150307 28300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.093363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.131:59719)
0925 01:17:29.125262 (+ 31899us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:29.125284 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:29.126533 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:29.127332 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:29.127345 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:29.128568 (+ 1223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:29.129667 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.129676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.131167 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.131172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.134298 (+ 3126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.134318 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.136873 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.136882 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.137207 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.137891 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:29.137907 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:29.141067 (+ 3160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:29.145870 (+ 4803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:29.145875 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:29.145879 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:29.146685 (+ 806us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:29.147200 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:29.147205 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:29.147208 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:29.147346 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:29.148597 (+ 1251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:29.148602 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:29.148797 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 01:17:29.148941 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 01:17:29.149197 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31628,"mutex_wait_us":1211,"thread_start_us":29985,"threads_started":1}
I20260925 01:17:29.150332 28301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.093229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:35253 (local address 127.20.102.131:54909)
0925 01:17:29.123684 (+ 30455us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:29.123707 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:29.126396 (+ 2689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:29.127336 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:29.127347 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:29.128543 (+ 1196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:29.129712 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.129739 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.131157 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.131162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.133274 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.133286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.134470 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.134477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.134769 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.135687 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:29.135717 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:29.139321 (+ 3604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:29.143532 (+ 4211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:29.143540 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:29.143570 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:29.144377 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:29.144892 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:29.144896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:29.144900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:29.145128 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:29.146358 (+ 1230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:29.146367 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:29.146821 (+ 454us) client_negotiation.cc:769] Sending connection context
0925 01:17:29.148264 (+ 1443us) client_negotiation.cc:240] Negotiation successful
0925 01:17:29.148827 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30285,"mutex_wait_us":292,"spinlock_wait_cycles":53888,"thread_start_us":31535,"threads_started":1}
I20260925 01:17:29.151044 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.092155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:59719 (local address 127.20.102.190:37687)
0925 01:17:29.124507 (+ 32352us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.124517 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.125448 (+ 931us) server_negotiation.cc:407] Connection header received
0925 01:17:29.126847 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.126855 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.126934 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.127036 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:29.129829 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.130979 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.134442 (+ 3463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.135861 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.135980 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.141298 (+ 5318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:29.141307 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:29.141314 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:29.141363 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:29.145738 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.146787 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.146795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.146801 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.146990 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.147441 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.147448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.147454 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.147852 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:29.148366 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.149101 (+ 735us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.149383 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32175,"thread_start_us":33260,"threads_started":1}
I20260925 01:17:29.152421 28294 heartbeater.cc:347] Connected to a master server at 127.20.102.189:35253
I20260925 01:17:29.152472 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.092485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:52371 (local address 127.20.102.188:39747)
0925 01:17:29.119423 (+ 26938us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.119432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.124057 (+ 4625us) server_negotiation.cc:407] Connection header received
0925 01:17:29.126853 (+ 2796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.126861 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.126957 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.127092 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:29.129870 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.130679 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.134040 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.134681 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.134776 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.142136 (+ 7360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:29.142145 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:29.142150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:29.142193 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:29.145854 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.146831 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.146838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.146844 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.147062 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.147954 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.147962 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.147968 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.148401 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:29.148539 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.151745 (+ 3206us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.152059 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26762,"thread_start_us":31369,"threads_started":1}
I20260925 01:17:29.152792 28294 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:29.153723 28294 heartbeater.cc:511] Master 127.20.102.189:35253 requested a full tablet report, sending...
I20260925 01:17:29.154703 28293 heartbeater.cc:347] Connected to a master server at 127.20.102.190:37687
I20260925 01:17:29.154825 28293 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:29.155320 28293 heartbeater.cc:511] Master 127.20.102.190:37687 requested a full tablet report, sending...
I20260925 01:17:29.154626 28298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.092727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:39747 (local address 127.20.102.131:52371)
0925 01:17:29.120200 (+ 27473us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:29.120299 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:29.126390 (+ 6091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:29.127332 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:29.127345 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:29.128568 (+ 1223us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:29.129643 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.129674 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.130850 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.130873 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.133647 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.133658 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.135072 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.135081 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.135256 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.135841 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:29.135860 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:29.141900 (+ 6040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:29.146009 (+ 4109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:29.146013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:29.146018 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:29.146460 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:29.147467 (+ 1007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:29.147471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:29.147475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:29.147608 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:29.151284 (+ 3676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:29.151291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:29.151459 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 01:17:29.153874 (+ 2415us) client_negotiation.cc:240] Negotiation successful
0925 01:17:29.154184 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27058,"mutex_wait_us":67,"spinlock_wait_cycles":16640,"thread_start_us":26803,"threads_started":1}
I20260925 01:17:29.155768 27633 ts_manager.cc:194] Registered new tserver with Master: 03178d57cd3d4370aa02d2a3dbcadf1a (127.20.102.131:36489)
I20260925 01:17:29.156354 27032 ts_manager.cc:194] Re-registered known tserver with Master: 03178d57cd3d4370aa02d2a3dbcadf1a (127.20.102.131:36489)
I20260925 01:17:29.157684 27032 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:59719
I20260925 01:17:29.158082 28292 heartbeater.cc:347] Connected to a master server at 127.20.102.188:39747
I20260925 01:17:29.158212 28292 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:29.158705 28292 heartbeater.cc:511] Master 127.20.102.188:39747 requested a full tablet report, sending...
I20260925 01:17:29.160056 27822 ts_manager.cc:194] Registered new tserver with Master: 03178d57cd3d4370aa02d2a3dbcadf1a (127.20.102.131:36489)
I20260925 01:17:29.173280 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.131824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.20.102.188:39747)
0925 01:17:29.153395 (+ 21571us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.153402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.153420 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:17:29.153467 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.153471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.153584 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.153683 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:29.158829 (+ 5146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.159886 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.161976 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.162657 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.162763 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.166921 (+ 4158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:29.166927 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:29.166932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:29.166967 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:29.170003 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.170861 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.170867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.170871 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.171032 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.171503 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.171508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.171513 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.171866 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:29.172127 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.172507 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.172790 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21465,"thread_start_us":32044,"threads_started":1}
I20260925 01:17:29.177073 28295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.135950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56070 (local address 127.20.102.190:37687)
0925 01:17:29.152246 (+ 16296us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.152254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.161651 (+ 9397us) server_negotiation.cc:407] Connection header received
0925 01:17:29.161827 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.161835 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.161948 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.162052 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:29.165125 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.166006 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.167731 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.168112 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.168240 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.171189 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:29.171198 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:29.171202 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:29.171237 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:29.174449 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.175158 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.175164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.175167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.175294 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.175651 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.175656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.175659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.175945 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:29.176203 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.176514 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.176761 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16158,"thread_start_us":25982,"threads_started":1}
I20260925 01:17:29.180805 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.140657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45770 (local address 127.20.102.189:35253)
0925 01:17:29.148205 (+ 7548us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.148210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.165013 (+ 16803us) server_negotiation.cc:407] Connection header received
0925 01:17:29.165280 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.165285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.165372 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.165466 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:29.167410 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.168485 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.170516 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.171115 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.171237 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.174643 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:29.174649 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:29.174653 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:29.174691 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:29.177593 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.178214 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.178220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.178223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.178346 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.179047 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.179067 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.179071 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.179478 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:29.179783 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.180184 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.180422 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7445,"thread_start_us":20197,"threads_started":1}
I20260925 01:17:29.181288 27032 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56070:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:17:29.188030 27032 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:17:29.230770 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.209363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.20.102.131:36489)
0925 01:17:29.221461 (+ 12098us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.221468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.222006 (+ 538us) server_negotiation.cc:407] Connection header received
0925 01:17:29.222221 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.222231 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.222568 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.223784 (+ 1216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:29.224138 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.226089 (+ 1951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.228291 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.229569 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.229747 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.229877 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.230290 (+ 413us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.230444 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11954,"thread_start_us":11786,"threads_started":1}
I20260925 01:17:29.230834 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.209081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:36489 (local address 127.0.0.1:55000)
0925 01:17:29.221874 (+ 12793us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:29.221896 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:29.222051 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:29.222743 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:29.222747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:29.222773 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:29.223199 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.223212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.226244 (+ 3032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.226250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.228142 (+ 1892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.228155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.229967 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.229972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.230097 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.230120 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 01:17:29.230223 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 01:17:29.230351 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12641,"mutex_wait_us":37,"thread_start_us":13355,"threads_started":1}
I20260925 01:17:29.231590 28320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.208360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:38049 (local address 127.0.0.1:51326)
0925 01:17:29.209066 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:29.209086 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:29.215512 (+ 6426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:29.223813 (+ 8301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:29.223821 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:29.223859 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:29.224340 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.224355 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.226256 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.226261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.228822 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.228834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.230714 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.230720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.230918 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.230968 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 01:17:29.231029 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 01:17:29.231144 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":85,"spinlock_wait_cycles":2304,"thread_start_us":394,"threads_started":1}
I20260925 01:17:29.232993 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.209532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.20.102.130:43921)
0925 01:17:29.220943 (+ 11411us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.220956 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.222777 (+ 1821us) server_negotiation.cc:407] Connection header received
0925 01:17:29.222934 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.222942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.223435 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.223635 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:29.224358 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.225711 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.229530 (+ 3819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.230918 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.231049 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.231207 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.232490 (+ 1283us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.232730 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10937,"thread_start_us":11153,"threads_started":1}
I20260925 01:17:29.233012 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.208646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51326 (local address 127.20.102.129:38049)
0925 01:17:29.223280 (+ 14634us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.223289 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.223315 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:17:29.223431 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.223436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.223606 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.223823 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:29.224551 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.226048 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.228991 (+ 2943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.230484 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.232298 (+ 1814us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.232483 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.232553 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.232728 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14468,"thread_start_us":14448,"threads_started":1}
I20260925 01:17:29.233235 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.209293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:43921 (local address 127.0.0.1:59174)
0925 01:17:29.222664 (+ 13371us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:29.222686 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:29.222798 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:29.223653 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:29.223658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:29.223681 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:29.224159 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.224172 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.225860 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.225866 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.229337 (+ 3471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.229351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.231472 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.231524 (+ 52us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.231712 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.231749 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 01:17:29.231844 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 01:17:29.231978 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13249,"mutex_wait_us":112,"spinlock_wait_cycles":2432,"thread_start_us":12829,"threads_started":1}
I20260925 01:17:29.234926 28227 tablet_service.cc:1467] Processing CreateTablet for tablet c684cc4f03a5447b85bdec0cf804788e (DEFAULT_TABLE table=test-table [id=bf50f3c9cd94482d8b54cebc4b10d8ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:29.236189 28227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c684cc4f03a5447b85bdec0cf804788e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:29.236590 27938 tablet_service.cc:1467] Processing CreateTablet for tablet c684cc4f03a5447b85bdec0cf804788e (DEFAULT_TABLE table=test-table [id=bf50f3c9cd94482d8b54cebc4b10d8ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:29.236590 28086 tablet_service.cc:1467] Processing CreateTablet for tablet c684cc4f03a5447b85bdec0cf804788e (DEFAULT_TABLE table=test-table [id=bf50f3c9cd94482d8b54cebc4b10d8ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:29.237900 28086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c684cc4f03a5447b85bdec0cf804788e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:29.237926 27938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c684cc4f03a5447b85bdec0cf804788e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:29.248077 28326 tablet_bootstrap.cc:492] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a: Bootstrap starting.
I20260925 01:17:29.251972 28326 tablet_bootstrap.cc:654] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:29.253237 28326 log.cc:824] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:29.253434 28327 tablet_bootstrap.cc:492] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656: Bootstrap starting.
I20260925 01:17:29.257375 28328 tablet_bootstrap.cc:492] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32: Bootstrap starting.
I20260925 01:17:29.257937 28327 tablet_bootstrap.cc:654] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:29.260366 28327 log.cc:824] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:29.260421 28328 tablet_bootstrap.cc:654] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:29.261795 28328 log.cc:824] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:29.263428 28326 tablet_bootstrap.cc:492] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a: No bootstrap required, opened a new log
I20260925 01:17:29.263528 28327 tablet_bootstrap.cc:492] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656: No bootstrap required, opened a new log
I20260925 01:17:29.263736 28326 ts_tablet_manager.cc:1397] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260925 01:17:29.263801 28327 ts_tablet_manager.cc:1397] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260925 01:17:29.264643 28328 tablet_bootstrap.cc:492] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32: No bootstrap required, opened a new log
I20260925 01:17:29.264943 28328 ts_tablet_manager.cc:1397] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 01:17:29.271103 28326 raft_consensus.cc:348] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } }
I20260925 01:17:29.271458 28326 raft_consensus.cc:374] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:29.271586 28326 raft_consensus.cc:729] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03178d57cd3d4370aa02d2a3dbcadf1a, State: Initialized, Role: FOLLOWER
I20260925 01:17:29.272199 28326 consensus_queue.cc:260] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } }
I20260925 01:17:29.273933 28326 ts_tablet_manager.cc:1428] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260925 01:17:29.274209 28293 heartbeater.cc:503] Master 127.20.102.190:37687 was elected leader, sending a full tablet report...
I20260925 01:17:29.274391 28327 raft_consensus.cc:348] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } }
I20260925 01:17:29.274793 28327 raft_consensus.cc:374] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:29.274890 28327 raft_consensus.cc:729] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6804c301ed464eaaa95340cae0f1b656, State: Initialized, Role: FOLLOWER
I20260925 01:17:29.275580 28327 consensus_queue.cc:260] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } }
I20260925 01:17:29.275630 28328 raft_consensus.cc:348] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } }
I20260925 01:17:29.276013 28328 raft_consensus.cc:374] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:29.276144 28328 raft_consensus.cc:729] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2338a1bec88f4c1d89f7b19f487fab32, State: Initialized, Role: FOLLOWER
I20260925 01:17:29.276947 28328 consensus_queue.cc:260] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } }
I20260925 01:17:29.279072 28328 ts_tablet_manager.cc:1428] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 01:17:29.279409 28152 heartbeater.cc:503] Master 127.20.102.190:37687 was elected leader, sending a full tablet report...
I20260925 01:17:29.287662 28327 ts_tablet_manager.cc:1428] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656: Time spent starting tablet: real 0.024s user 0.020s sys 0.003s
I20260925 01:17:29.314626 28334 raft_consensus.cc:482] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:17:29.314920 28334 raft_consensus.cc:504] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } }
I20260925 01:17:29.316615 28334 leader_election.cc:288] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6804c301ed464eaaa95340cae0f1b656 (127.20.102.129:38049), 03178d57cd3d4370aa02d2a3dbcadf1a (127.20.102.131:36489)
I20260925 01:17:29.325819 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.317057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:45747 (local address 127.20.102.129:38049)
0925 01:17:29.317313 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.317322 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.318112 (+ 790us) server_negotiation.cc:407] Connection header received
0925 01:17:29.318343 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.318348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.318447 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.318552 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:29.319179 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.320313 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.323433 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.324763 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.324909 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.325017 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.325410 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.325565 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:17:29.326054 28336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.316887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:38049 (local address 127.20.102.130:45747)
0925 01:17:29.318010 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:29.318028 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:29.318185 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:29.318581 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:29.318588 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:29.318630 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:29.319025 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.319052 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.320473 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.320479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.323243 (+ 2764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.323256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.325001 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.325008 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.325236 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.325277 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 01:17:29.325409 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 01:17:29.325613 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":231,"thread_start_us":801,"threads_started":1}
I20260925 01:17:29.326232 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.317133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:36413 (local address 127.20.102.131:36489)
0925 01:17:29.317368 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.317374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.318169 (+ 795us) server_negotiation.cc:407] Connection header received
0925 01:17:29.318439 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.318448 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.318524 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.318617 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:29.319304 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.320343 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.323433 (+ 3090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.324750 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.324909 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.324984 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.325579 (+ 595us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.325711 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 01:17:29.326898 28335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.317102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:36489 (local address 127.20.102.130:36413)
0925 01:17:29.318082 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:29.318097 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:29.318253 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:29.318790 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:29.318794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:29.318813 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:29.319144 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.319156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.320473 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.320479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.323241 (+ 2762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:29.323256 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.325225 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.325231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:29.325392 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.325425 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:17:29.326409 (+ 984us) client_negotiation.cc:240] Negotiation successful
0925 01:17:29.326562 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"mutex_wait_us":189,"thread_start_us":492,"threads_started":1}
I20260925 01:17:29.327198 27958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c684cc4f03a5447b85bdec0cf804788e" candidate_uuid: "2338a1bec88f4c1d89f7b19f487fab32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6804c301ed464eaaa95340cae0f1b656" is_pre_election: true
I20260925 01:17:29.327726 27958 raft_consensus.cc:2454] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2338a1bec88f4c1d89f7b19f487fab32 in term 0.
I20260925 01:17:29.328157 28247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c684cc4f03a5447b85bdec0cf804788e" candidate_uuid: "2338a1bec88f4c1d89f7b19f487fab32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" is_pre_election: true
I20260925 01:17:29.328466 28041 leader_election.cc:302] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2338a1bec88f4c1d89f7b19f487fab32, 6804c301ed464eaaa95340cae0f1b656; no voters:
I20260925 01:17:29.328800 28247 raft_consensus.cc:2454] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2338a1bec88f4c1d89f7b19f487fab32 in term 0.
I20260925 01:17:29.328928 28334 raft_consensus.cc:2781] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:17:29.329075 28334 raft_consensus.cc:482] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:17:29.329164 28334 raft_consensus.cc:3037] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:29.332290 28334 raft_consensus.cc:504] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } }
I20260925 01:17:29.332914 28334 leader_election.cc:288] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [CANDIDATE]: Term 1 election: Requested vote from peers 6804c301ed464eaaa95340cae0f1b656 (127.20.102.129:38049), 03178d57cd3d4370aa02d2a3dbcadf1a (127.20.102.131:36489)
I20260925 01:17:29.333293 27958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c684cc4f03a5447b85bdec0cf804788e" candidate_uuid: "2338a1bec88f4c1d89f7b19f487fab32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6804c301ed464eaaa95340cae0f1b656"
I20260925 01:17:29.333339 28247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c684cc4f03a5447b85bdec0cf804788e" candidate_uuid: "2338a1bec88f4c1d89f7b19f487fab32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a"
I20260925 01:17:29.333496 27958 raft_consensus.cc:3037] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:29.333521 28247 raft_consensus.cc:3037] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:29.335616 27958 raft_consensus.cc:2454] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2338a1bec88f4c1d89f7b19f487fab32 in term 1.
I20260925 01:17:29.335808 28247 raft_consensus.cc:2454] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2338a1bec88f4c1d89f7b19f487fab32 in term 1.
I20260925 01:17:29.336017 28041 leader_election.cc:302] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2338a1bec88f4c1d89f7b19f487fab32, 6804c301ed464eaaa95340cae0f1b656; no voters:
I20260925 01:17:29.336298 28334 raft_consensus.cc:2781] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:29.336849 28334 raft_consensus.cc:686] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [term 1 LEADER]: Becoming Leader. State: Replica: 2338a1bec88f4c1d89f7b19f487fab32, State: Running, Role: LEADER
I20260925 01:17:29.337420 28334 consensus_queue.cc:237] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } }
I20260925 01:17:29.343732 27032 catalog_manager.cc:5495] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2338a1bec88f4c1d89f7b19f487fab32 (127.20.102.130). New cstate: current_term: 1 leader_uuid: "2338a1bec88f4c1d89f7b19f487fab32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } health_report { overall_health: HEALTHY } } }
W20260925 01:17:29.376885 28297 tablet.cc:2369] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:17:29.386466 28325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.380588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.20.102.130:43921)
0925 01:17:29.380822 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.380828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.380850 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:17:29.380975 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.380981 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.381094 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.381243 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:17:29.381553 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.382645 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.383907 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.384548 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.384669 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.385220 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:17:29.385891 (+ 671us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:17:29.386046 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.386091 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.386171 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260925 01:17:29.394543 28247 raft_consensus.cc:1261] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a [term 1 FOLLOWER]: Refusing update from remote peer 2338a1bec88f4c1d89f7b19f487fab32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 01:17:29.394586 28006 tablet.cc:2369] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:17:29.394699 27958 raft_consensus.cc:1261] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656 [term 1 FOLLOWER]: Refusing update from remote peer 2338a1bec88f4c1d89f7b19f487fab32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:29.395632 28337 consensus_queue.cc:1035] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:29.396059 28334 consensus_queue.cc:1035] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:29.408521 28340 mvcc.cc:205] Tried to move back new op lower bound from 7333064906310119424 to 7333064906089975808. Current Snapshot: MvccSnapshot[applied={T|T < 7333064906310119424}]
I20260925 01:17:29.426625 27033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:29.426896 27033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56070:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:29.434443 27033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 874aa79fff1748e58a20f66897829f27: Sending DeleteTablet for 3 replicas of tablet c684cc4f03a5447b85bdec0cf804788e
I20260925 01:17:29.435462 27938 tablet_service.cc:1514] Processing DeleteTablet for tablet c684cc4f03a5447b85bdec0cf804788e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:29 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:51326
I20260925 01:17:29.435652 28227 tablet_service.cc:1514] Processing DeleteTablet for tablet c684cc4f03a5447b85bdec0cf804788e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:29 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:55000
I20260925 01:17:29.436723 28086 tablet_service.cc:1514] Processing DeleteTablet for tablet c684cc4f03a5447b85bdec0cf804788e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:29 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:59174
I20260925 01:17:29.437631 28351 tablet_replica.cc:330] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32: stopping tablet replica
I20260925 01:17:29.438122 28351 raft_consensus.cc:2229] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:17:29.438871 28351 raft_consensus.cc:2258] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:29.441087 28351 ts_tablet_manager.cc:1905] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:29.449954 28351 ts_tablet_manager.cc:1918] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:29.450206 28351 log.cc:1196] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/c684cc4f03a5447b85bdec0cf804788e
I20260925 01:17:29.450978 28351 ts_tablet_manager.cc:1939] T c684cc4f03a5447b85bdec0cf804788e P 2338a1bec88f4c1d89f7b19f487fab32: Deleting consensus metadata
I20260925 01:17:29.452963 27017 catalog_manager.cc:4841] TS 2338a1bec88f4c1d89f7b19f487fab32 (127.20.102.130:43921): tablet c684cc4f03a5447b85bdec0cf804788e (table test-table [id=bf50f3c9cd94482d8b54cebc4b10d8ee]) successfully deleted
I20260925 01:17:29.457921 28350 tablet_replica.cc:330] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a: stopping tablet replica
I20260925 01:17:29.458849 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 27001
I20260925 01:17:29.459877 27063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:29.461938 28350 raft_consensus.cc:2229] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:29.462430 28350 raft_consensus.cc:2258] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:29.465135 28350 ts_tablet_manager.cc:1905] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:29.465684 28349 tablet_replica.cc:330] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656: stopping tablet replica
I20260925 01:17:29.466125 28349 raft_consensus.cc:2229] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:29.466694 28349 raft_consensus.cc:2258] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:29.468997 28349 ts_tablet_manager.cc:1905] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:29.475611 28350 ts_tablet_manager.cc:1918] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:29.476171 28350 log.cc:1196] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/c684cc4f03a5447b85bdec0cf804788e
I20260925 01:17:29.476181 28349 ts_tablet_manager.cc:1918] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:29.476351 28349 log.cc:1196] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/c684cc4f03a5447b85bdec0cf804788e
I20260925 01:17:29.476863 28350 ts_tablet_manager.cc:1939] T c684cc4f03a5447b85bdec0cf804788e P 03178d57cd3d4370aa02d2a3dbcadf1a: Deleting consensus metadata
I20260925 01:17:29.477025 28349 ts_tablet_manager.cc:1939] T c684cc4f03a5447b85bdec0cf804788e P 6804c301ed464eaaa95340cae0f1b656: Deleting consensus metadata
I20260925 01:17:29.478178 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.438147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45772 (local address 127.20.102.189:35253)
0925 01:17:29.438313 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.438319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.457561 (+ 19242us) server_negotiation.cc:407] Connection header received
0925 01:17:29.457633 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.457640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.457725 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.457821 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:29.461542 (+ 3721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.462582 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.465200 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.465841 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.465966 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.470539 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:29.470545 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:29.470549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:29.470587 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:29.474508 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.475415 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.475420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.475424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.475580 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.475988 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.475994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.475997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.476888 (+ 891us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:29.477307 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.477602 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.477807 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:17:29.494834 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.457453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.20.102.188:39747)
0925 01:17:29.457684 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.457692 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.478174 (+ 20482us) server_negotiation.cc:407] Connection header received
0925 01:17:29.478374 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.478380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.478511 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.478634 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:29.480053 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.481315 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.483231 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.483910 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.484039 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.487548 (+ 3509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:29.487555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:29.487559 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:29.487599 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:29.490942 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.491745 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.491752 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.491755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.491913 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.492349 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.492355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.492360 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.492748 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:29.493021 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.494175 (+ 1154us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.494427 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:17:29.601637 27018 catalog_manager.cc:4841] TS 03178d57cd3d4370aa02d2a3dbcadf1a (127.20.102.131:36489): tablet c684cc4f03a5447b85bdec0cf804788e (table test-table [id=bf50f3c9cd94482d8b54cebc4b10d8ee]) successfully deleted
I20260925 01:17:29.601671 27019 catalog_manager.cc:4841] TS 6804c301ed464eaaa95340cae0f1b656 (127.20.102.129:38049): tablet c684cc4f03a5447b85bdec0cf804788e (table test-table [id=bf50f3c9cd94482d8b54cebc4b10d8ee]) successfully deleted
I20260925 01:17:29.602365 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 27001
I20260925 01:17:29.662209 28299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.631459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.20.102.189:35253)
0925 01:17:29.631695 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.631703 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.641557 (+ 9854us) server_negotiation.cc:407] Connection header received
0925 01:17:29.641952 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.641961 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.642069 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.642213 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:29.648247 (+ 6034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.649328 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.651175 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.651914 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.652035 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.655592 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:29.655598 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:29.655603 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:29.655647 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:29.659181 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.660068 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.660076 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.660082 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.660278 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.660655 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.660662 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.660665 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.660982 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:29.661278 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.661626 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.661846 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:17:29.666221 28296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:29.633900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.20.102.188:39747)
0925 01:17:29.634153 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:29.634163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:29.650888 (+ 16725us) server_negotiation.cc:407] Connection header received
0925 01:17:29.651169 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:29.651175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:29.651266 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:29.651335 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:29.653521 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.654261 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.655614 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:29.656315 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:29.656454 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:29.659506 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:29.659511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:29.659514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:29.659547 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:29.663640 (+ 4093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.664228 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.664251 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.664255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.664398 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:29.664822 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:29.664826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:29.664830 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:29.665093 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:29.665354 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:29.665678 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 01:17:29.665913 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:17:30.419363 28361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:30.418687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.130:37507)
0925 01:17:30.419083 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:30.419203 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":196,"threads_started":1}
W20260925 01:17:30.419777 28152 heartbeater.cc:644] Failed to heartbeat to 127.20.102.190:37687 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
I20260925 01:17:30.420876 28362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:30.420096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.129:57075)
0925 01:17:30.420550 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:30.420654 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":242,"threads_started":1}
W20260925 01:17:30.421265 28004 heartbeater.cc:644] Failed to heartbeat to 127.20.102.190:37687 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
I20260925 01:17:30.427803 28363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:30.427040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.131:47891)
0925 01:17:30.427504 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:30.427637 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":236,"threads_started":1}
W20260925 01:17:30.428179 28293 heartbeater.cc:644] Failed to heartbeat to 127.20.102.190:37687 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
I20260925 01:17:30.933351 28364 raft_consensus.cc:482] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 874aa79fff1748e58a20f66897829f27)
I20260925 01:17:30.933535 28364 raft_consensus.cc:504] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } }
I20260925 01:17:30.934279 28364 leader_election.cc:288] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 874aa79fff1748e58a20f66897829f27 (127.20.102.190:37687), 4281250057b94e258de2aac45dd6870f (127.20.102.188:39747)
I20260925 01:17:30.945765 28367 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 874aa79fff1748e58a20f66897829f27)
I20260925 01:17:30.946067 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:30.934384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56470)
0925 01:17:30.945225 (+ 10841us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:30.945564 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10622,"thread_start_us":10122,"threads_started":1}
W20260925 01:17:30.946596 27620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
I20260925 01:17:30.946785 28367 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } }
I20260925 01:17:30.951937 28367 leader_election.cc:288] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 874aa79fff1748e58a20f66897829f27 (127.20.102.190:37687), e99c1f0b8d2a428784076443be082d04 (127.20.102.189:35253)
I20260925 01:17:30.954957 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:30.948022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56474)
0925 01:17:30.954293 (+ 6271us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:30.954663 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6092,"threads_started":1}
W20260925 01:17:30.956046 27809 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
I20260925 01:17:30.957229 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:30.950767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56482)
0925 01:17:30.956944 (+ 6177us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:30.957044 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6032}
W20260925 01:17:30.957470 27620 leader_election.cc:334] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 874aa79fff1748e58a20f66897829f27 (127.20.102.190:37687): Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
I20260925 01:17:30.958382 28372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:30.957682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56492)
0925 01:17:30.957876 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:30.958114 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 01:17:30.958618 27809 leader_election.cc:334] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 874aa79fff1748e58a20f66897829f27 (127.20.102.190:37687): Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
I20260925 01:17:30.960214 28366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:30.934453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:39747 (local address 127.0.0.1:41806)
0925 01:17:30.946386 (+ 11933us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:30.946403 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:30.946540 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:30.951075 (+ 4535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:30.951108 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:30.951144 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:30.951622 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:30.951636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:30.955562 (+ 3926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:30.955566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:30.957899 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:30.957909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:30.959668 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:30.959673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:30.959796 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:30.959821 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 01:17:30.959889 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 01:17:30.959973 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11816,"mutex_wait_us":48,"thread_start_us":11599,"threads_started":1}
I20260925 01:17:30.960485 28368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:30.934694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.20.102.188:39747)
0925 01:17:30.945410 (+ 10716us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:30.945418 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:30.946490 (+ 1072us) server_negotiation.cc:407] Connection header received
0925 01:17:30.946690 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:30.946695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:30.946938 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:30.947227 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:30.951814 (+ 4587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:30.955361 (+ 3547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:30.958061 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:30.959475 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:30.959656 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:30.959727 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:30.959953 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 01:17:30.960103 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":10604,"thread_start_us":10721,"threads_started":1}
I20260925 01:17:30.961109 27832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e99c1f0b8d2a428784076443be082d04" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4281250057b94e258de2aac45dd6870f" is_pre_election: true
I20260925 01:17:30.961444 27832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e99c1f0b8d2a428784076443be082d04 in term 1.
I20260925 01:17:30.961932 27617 leader_election.cc:302] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4281250057b94e258de2aac45dd6870f, e99c1f0b8d2a428784076443be082d04; no voters: 874aa79fff1748e58a20f66897829f27
I20260925 01:17:30.962266 28364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 01:17:30.962437 28364 raft_consensus.cc:482] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 874aa79fff1748e58a20f66897829f27)
I20260925 01:17:30.962580 28364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:17:30.962622 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:30.948022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:35253 (local address 127.0.0.1:45792)
0925 01:17:30.955156 (+ 7134us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:30.955172 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:30.955361 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:30.955984 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:30.955990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:30.956022 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:30.956331 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:30.956343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:30.957875 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:30.957879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:30.960148 (+ 2269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:30.960161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:30.961867 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:30.961873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:30.961998 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:30.962019 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 01:17:30.962122 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 01:17:30.962270 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7032,"mutex_wait_us":486,"thread_start_us":8244,"threads_started":1}
I20260925 01:17:30.962723 28371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:30.948214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45792 (local address 127.20.102.189:35253)
0925 01:17:30.952663 (+ 4449us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:30.952670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:30.955302 (+ 2632us) server_negotiation.cc:407] Connection header received
0925 01:17:30.955490 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:30.955495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:30.955568 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:30.955670 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:30.956523 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:30.957770 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:30.960448 (+ 2678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:30.961683 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:30.961852 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:30.961928 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:30.962173 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 01:17:30.962323 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4308,"thread_start_us":4279,"threads_started":1}
I20260925 01:17:30.963191 27642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4281250057b94e258de2aac45dd6870f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e99c1f0b8d2a428784076443be082d04" is_pre_election: true
I20260925 01:17:30.965142 28364 raft_consensus.cc:504] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } }
I20260925 01:17:30.965600 27642 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4281250057b94e258de2aac45dd6870f in current term 2: Already voted for candidate e99c1f0b8d2a428784076443be082d04 in this term.
I20260925 01:17:30.965827 28364 leader_election.cc:288] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [CANDIDATE]: Term 2 election: Requested vote from peers 874aa79fff1748e58a20f66897829f27 (127.20.102.190:37687), 4281250057b94e258de2aac45dd6870f (127.20.102.188:39747)
I20260925 01:17:30.966276 27832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e99c1f0b8d2a428784076443be082d04" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4281250057b94e258de2aac45dd6870f"
I20260925 01:17:30.966540 27832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:17:30.966553 28366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:30.966173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56500)
0925 01:17:30.966318 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:30.966403 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 01:17:30.966576 27808 leader_election.cc:302] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4281250057b94e258de2aac45dd6870f; no voters: 874aa79fff1748e58a20f66897829f27, e99c1f0b8d2a428784076443be082d04
I20260925 01:17:30.967468 28366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:30.967129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56504)
0925 01:17:30.967285 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:30.967357 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 01:17:30.967636 27620 leader_election.cc:334] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 874aa79fff1748e58a20f66897829f27 (127.20.102.190:37687): Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
I20260925 01:17:30.968928 27832 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e99c1f0b8d2a428784076443be082d04 in term 2.
I20260925 01:17:30.969123 28367 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 01:17:30.969314 27617 leader_election.cc:302] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4281250057b94e258de2aac45dd6870f, e99c1f0b8d2a428784076443be082d04; no voters: 874aa79fff1748e58a20f66897829f27
I20260925 01:17:30.969630 28364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:17:30.970125 28364 raft_consensus.cc:686] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [term 2 LEADER]: Becoming Leader. State: Replica: e99c1f0b8d2a428784076443be082d04, State: Running, Role: LEADER
I20260925 01:17:30.970490 28364 consensus_queue.cc:237] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } }
I20260925 01:17:30.972289 28374 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e99c1f0b8d2a428784076443be082d04. Latest consensus state: current_term: 2 leader_uuid: "e99c1f0b8d2a428784076443be082d04" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } } }
I20260925 01:17:30.972713 28374 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:30.973304 28376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:17:30.976642 28376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:17:30.977542 28376 catalog_manager.cc:1259] Loaded cluster ID: 6a6d5cf20145481db80f59d94174d925
I20260925 01:17:30.977639 28376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:17:30.978765 28376 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:17:30.979542 28376 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: Loaded TSK: 0
I20260925 01:17:30.980363 28376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:17:30.991935 27633 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:17:31.000105 27633 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:17:31.004608 28366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.004180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56532)
0925 01:17:31.004351 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.004464 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 01:17:31.004745 27832 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [term 2 FOLLOWER]: Refusing update from remote peer e99c1f0b8d2a428784076443be082d04: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 01:17:31.005395 28374 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 01:17:31.005780 28366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.005342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56538)
0925 01:17:31.005549 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.005613 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 01:17:31.006292 27620 consensus_peers.cc:487] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 -> Peer 874aa79fff1748e58a20f66897829f27 (127.20.102.190:37687): Couldn't send request to peer 874aa79fff1748e58a20f66897829f27. Status: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 01:17:31.008626 28367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e99c1f0b8d2a428784076443be082d04. Latest consensus state: current_term: 2 leader_uuid: "e99c1f0b8d2a428784076443be082d04" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } } }
I20260925 01:17:31.009073 28367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:17:31.010121 28377 mvcc.cc:205] Tried to move back new op lower bound from 7333064912908165120 to 7333064912778719232. Current Snapshot: MvccSnapshot[applied={T|T < 7333064912908165120}]
I20260925 01:17:31.010799 28374 sys_catalog.cc:455] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e99c1f0b8d2a428784076443be082d04" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } } }
I20260925 01:17:31.011094 28374 sys_catalog.cc:458] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:31.011888 28367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e99c1f0b8d2a428784076443be082d04" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "874aa79fff1748e58a20f66897829f27" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37687 } } peers { permanent_uuid: "e99c1f0b8d2a428784076443be082d04" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 35253 } attrs { promote: false } } peers { permanent_uuid: "4281250057b94e258de2aac45dd6870f" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 39747 } attrs { promote: false } } }
I20260925 01:17:31.012189 28367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4281250057b94e258de2aac45dd6870f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:17:31.041596 28366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.019640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:38049 (local address 127.0.0.1:51328)
0925 01:17:31.019865 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.019888 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:31.019996 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:31.033491 (+ 13495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:31.033496 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:31.033566 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:31.033833 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:31.033848 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.035877 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.035886 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:31.039024 (+ 3138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:31.039041 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.040774 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.040780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:31.040970 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:31.041016 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 01:17:31.041124 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 01:17:31.041264 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 01:17:31.041643 28383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.020167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.20.102.129:38049)
0925 01:17:31.032968 (+ 12801us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:31.032978 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:31.033013 (+ 35us) server_negotiation.cc:407] Connection header received
0925 01:17:31.033104 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:31.033113 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:31.033213 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:31.034011 (+ 798us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:31.034229 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.035667 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.039217 (+ 3550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.040547 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.040749 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:31.040838 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:31.041168 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 01:17:31.041318 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12628,"thread_start_us":13137,"threads_started":1}
I20260925 01:17:31.043058 27938 tablet_service.cc:1467] Processing CreateTablet for tablet 0d436e1d89e64046928ef5a8027e0c08 (DEFAULT_TABLE table=test-table [id=7fec254bca84410e9ffa690634f65e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:31.043537 27938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d436e1d89e64046928ef5a8027e0c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:31.043581 28365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.020043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:43921 (local address 127.0.0.1:59190)
0925 01:17:31.020428 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.020445 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:31.020585 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:31.038090 (+ 17505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:31.038095 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:31.038122 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:31.038417 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:31.038425 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.039544 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.039550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:31.041240 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:31.041252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.042787 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.042794 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:31.042967 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:31.043000 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:17:31.043113 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 01:17:31.043260 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"spinlock_wait_cycles":21120}
I20260925 01:17:31.043627 28384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.020411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.20.102.130:43921)
0925 01:17:31.037789 (+ 17378us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:31.037793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:31.037811 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:17:31.037872 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:31.037875 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:31.037943 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:31.038075 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:31.038607 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.039394 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.041420 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.042481 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.042729 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:31.042816 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:31.043142 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 01:17:31.043290 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17258,"thread_start_us":17157,"threads_started":1}
I20260925 01:17:31.044436 28086 tablet_service.cc:1467] Processing CreateTablet for tablet 0d436e1d89e64046928ef5a8027e0c08 (DEFAULT_TABLE table=test-table [id=7fec254bca84410e9ffa690634f65e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:31.044955 28086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d436e1d89e64046928ef5a8027e0c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:31.051045 28387 tablet_bootstrap.cc:492] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656: Bootstrap starting.
I20260925 01:17:31.052237 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.022698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:36489 (local address 127.0.0.1:55004)
0925 01:17:31.038035 (+ 15337us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.038053 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:31.038173 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:31.039804 (+ 1631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:31.039809 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:31.039833 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:31.040323 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:31.040335 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.041941 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.041946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:31.048551 (+ 6605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:31.048603 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.051310 (+ 2707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.051318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:31.051508 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:31.051543 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:17:31.051631 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 01:17:31.051776 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15208,"spinlock_wait_cycles":768,"thread_start_us":14425,"threads_started":1}
I20260925 01:17:31.052414 28388 tablet_bootstrap.cc:492] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32: Bootstrap starting.
I20260925 01:17:31.053179 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.023258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.20.102.131:36489)
0925 01:17:31.039240 (+ 15982us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:31.039247 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:31.039267 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:17:31.039337 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:31.039342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:31.039428 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:31.039539 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:31.040542 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.041766 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.049019 (+ 7253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.050899 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.051100 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:31.052611 (+ 1511us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:31.052671 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 01:17:31.052848 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15851,"thread_start_us":16379,"threads_started":1}
I20260925 01:17:31.053994 28227 tablet_service.cc:1467] Processing CreateTablet for tablet 0d436e1d89e64046928ef5a8027e0c08 (DEFAULT_TABLE table=test-table [id=7fec254bca84410e9ffa690634f65e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:31.054270 28388 tablet_bootstrap.cc:654] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:31.054503 28227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d436e1d89e64046928ef5a8027e0c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:31.054728 28387 tablet_bootstrap.cc:654] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:31.056792 28388 tablet_bootstrap.cc:492] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32: No bootstrap required, opened a new log
I20260925 01:17:31.056941 28388 ts_tablet_manager.cc:1397] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 01:17:31.057027 28387 tablet_bootstrap.cc:492] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656: No bootstrap required, opened a new log
I20260925 01:17:31.057184 28387 ts_tablet_manager.cc:1397] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 01:17:31.057760 28388 raft_consensus.cc:348] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } }
I20260925 01:17:31.057982 28388 raft_consensus.cc:374] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:31.058099 28388 raft_consensus.cc:729] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2338a1bec88f4c1d89f7b19f487fab32, State: Initialized, Role: FOLLOWER
I20260925 01:17:31.057938 28387 raft_consensus.cc:348] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } }
I20260925 01:17:31.058238 28387 raft_consensus.cc:374] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:31.058351 28387 raft_consensus.cc:729] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6804c301ed464eaaa95340cae0f1b656, State: Initialized, Role: FOLLOWER
I20260925 01:17:31.058353 28388 consensus_queue.cc:260] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } }
I20260925 01:17:31.058632 28387 consensus_queue.cc:260] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } }
I20260925 01:17:31.060020 28388 ts_tablet_manager.cc:1428] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 01:17:31.063051 28153 heartbeater.cc:503] Master 127.20.102.189:35253 was elected leader, sending a full tablet report...
I20260925 01:17:31.063644 28387 ts_tablet_manager.cc:1428] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260925 01:17:31.067006 28005 heartbeater.cc:503] Master 127.20.102.189:35253 was elected leader, sending a full tablet report...
I20260925 01:17:31.067615 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.063884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.129:41527)
0925 01:17:31.067124 (+ 3240us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.067317 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3064,"thread_start_us":2797,"threads_started":1}
I20260925 01:17:31.071383 28394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.068262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.129:37501)
0925 01:17:31.070987 (+ 2725us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.071139 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2579,"threads_started":1}
I20260925 01:17:31.074004 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.060412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.130:37831)
0925 01:17:31.073520 (+ 13108us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.073676 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12928,"thread_start_us":12799,"threads_started":1}
I20260925 01:17:31.074401 28395 tablet_bootstrap.cc:492] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a: Bootstrap starting.
I20260925 01:17:31.074513 28391 raft_consensus.cc:482] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:17:31.074717 28391 raft_consensus.cc:504] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } }
I20260925 01:17:31.075357 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.074762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.130:35561)
0925 01:17:31.075083 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.075195 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208}
I20260925 01:17:31.075845 28391 leader_election.cc:288] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6804c301ed464eaaa95340cae0f1b656 (127.20.102.129:38049), 03178d57cd3d4370aa02d2a3dbcadf1a (127.20.102.131:36489)
I20260925 01:17:31.076081 28247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d436e1d89e64046928ef5a8027e0c08" candidate_uuid: "2338a1bec88f4c1d89f7b19f487fab32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" is_pre_election: true
I20260925 01:17:31.076135 27958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d436e1d89e64046928ef5a8027e0c08" candidate_uuid: "2338a1bec88f4c1d89f7b19f487fab32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6804c301ed464eaaa95340cae0f1b656" is_pre_election: true
I20260925 01:17:31.076360 27958 raft_consensus.cc:2454] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2338a1bec88f4c1d89f7b19f487fab32 in term 0.
I20260925 01:17:31.076411 28395 tablet_bootstrap.cc:654] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:31.076901 28041 leader_election.cc:302] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2338a1bec88f4c1d89f7b19f487fab32, 6804c301ed464eaaa95340cae0f1b656; no voters:
W20260925 01:17:31.077139 28040 leader_election.cc:341] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 03178d57cd3d4370aa02d2a3dbcadf1a (127.20.102.131:36489): Illegal state: must be running to vote when last-logged opid is not known
I20260925 01:17:31.077320 28391 raft_consensus.cc:2781] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:17:31.077472 28391 raft_consensus.cc:482] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:17:31.077610 28391 raft_consensus.cc:3037] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:31.081516 28391 raft_consensus.cc:504] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } }
I20260925 01:17:31.082082 28395 tablet_bootstrap.cc:492] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a: No bootstrap required, opened a new log
I20260925 01:17:31.082240 28395 ts_tablet_manager.cc:1397] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260925 01:17:31.082365 28391 leader_election.cc:288] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [CANDIDATE]: Term 1 election: Requested vote from peers 6804c301ed464eaaa95340cae0f1b656 (127.20.102.129:38049), 03178d57cd3d4370aa02d2a3dbcadf1a (127.20.102.131:36489)
I20260925 01:17:31.082604 27958 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d436e1d89e64046928ef5a8027e0c08" candidate_uuid: "2338a1bec88f4c1d89f7b19f487fab32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6804c301ed464eaaa95340cae0f1b656"
I20260925 01:17:31.082823 27958 raft_consensus.cc:3037] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:31.082831 28247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d436e1d89e64046928ef5a8027e0c08" candidate_uuid: "2338a1bec88f4c1d89f7b19f487fab32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a"
I20260925 01:17:31.083055 28395 raft_consensus.cc:348] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } }
I20260925 01:17:31.083307 28395 raft_consensus.cc:374] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:31.083443 28395 raft_consensus.cc:729] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03178d57cd3d4370aa02d2a3dbcadf1a, State: Initialized, Role: FOLLOWER
I20260925 01:17:31.083724 28395 consensus_queue.cc:260] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } }
I20260925 01:17:31.084079 28247 raft_consensus.cc:3037] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:31.084769 28395 ts_tablet_manager.cc:1428] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 01:17:31.086061 27958 raft_consensus.cc:2454] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2338a1bec88f4c1d89f7b19f487fab32 in term 1.
I20260925 01:17:31.086453 28041 leader_election.cc:302] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2338a1bec88f4c1d89f7b19f487fab32, 6804c301ed464eaaa95340cae0f1b656; no voters:
I20260925 01:17:31.086624 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.085285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.131:35595)
0925 01:17:31.086125 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.086321 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":550,"threads_started":1}
I20260925 01:17:31.086772 28391 raft_consensus.cc:2781] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:31.087626 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.087277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.131:41003)
0925 01:17:31.087426 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.087515 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 01:17:31.087702 28247 raft_consensus.cc:2454] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2338a1bec88f4c1d89f7b19f487fab32 in term 1.
I20260925 01:17:31.089563 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.089069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.130:58969)
0925 01:17:31.089248 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.089363 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 01:17:31.090772 28294 heartbeater.cc:503] Master 127.20.102.189:35253 was elected leader, sending a full tablet report...
I20260925 01:17:31.096767 28391 raft_consensus.cc:686] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [term 1 LEADER]: Becoming Leader. State: Replica: 2338a1bec88f4c1d89f7b19f487fab32, State: Running, Role: LEADER
I20260925 01:17:31.097043 28391 consensus_queue.cc:237] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } }
I20260925 01:17:31.099639 27632 catalog_manager.cc:5495] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2338a1bec88f4c1d89f7b19f487fab32 (127.20.102.130). New cstate: current_term: 1 leader_uuid: "2338a1bec88f4c1d89f7b19f487fab32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2338a1bec88f4c1d89f7b19f487fab32" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 } health_report { overall_health: UNKNOWN } } }
I20260925 01:17:31.128250 28384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.123318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59202 (local address 127.20.102.130:43921)
0925 01:17:31.123512 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:31.123517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:31.123534 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:17:31.123670 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:31.123673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:31.123737 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:31.123809 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:17:31.124220 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.125053 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.126599 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.127063 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.127159 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:31.127612 (+ 453us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:17:31.127718 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:17:31.127783 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:31.127911 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 01:17:31.127969 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:17:31.131194 28247 raft_consensus.cc:1261] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a [term 1 FOLLOWER]: Refusing update from remote peer 2338a1bec88f4c1d89f7b19f487fab32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:31.131304 27958 raft_consensus.cc:1261] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656 [term 1 FOLLOWER]: Refusing update from remote peer 2338a1bec88f4c1d89f7b19f487fab32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:31.131954 28391 consensus_queue.cc:1035] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03178d57cd3d4370aa02d2a3dbcadf1a" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:31.132226 28400 consensus_queue.cc:1035] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6804c301ed464eaaa95340cae0f1b656" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 38049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:31.135063 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.134603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.131:45121)
0925 01:17:31.134834 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.134922 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260925 01:17:31.135088 28396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.134040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.129:32909)
0925 01:17:31.134445 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.134743 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260925 01:17:31.141963 28404 mvcc.cc:205] Tried to move back new op lower bound from 7333064913427632128 to 7333064913295495168. Current Snapshot: MvccSnapshot[applied={T|T < 7333064913427632128}]
I20260925 01:17:31.143290 28396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.142798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.129:32925)
0925 01:17:31.143032 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.143127 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260925 01:17:31.143687 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.142893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.131:50539)
0925 01:17:31.143436 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.143525 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429}
I20260925 01:17:31.144110 28405 mvcc.cc:205] Tried to move back new op lower bound from 7333064913427632128 to 7333064913295495168. Current Snapshot: MvccSnapshot[applied={T|T < 7333064913427632128}]
I20260925 01:17:31.145346 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.144569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.130:47857)
0925 01:17:31.145061 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.145168 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355}
I20260925 01:17:31.149139 27633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:31.149456 27633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:31.149739 28393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.149236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.20.102.130:55403)
0925 01:17:31.149456 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.149540 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260925 01:17:31.159593 27633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e99c1f0b8d2a428784076443be082d04: Sending DeleteTablet for 3 replicas of tablet 0d436e1d89e64046928ef5a8027e0c08
I20260925 01:17:31.160440 27938 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d436e1d89e64046928ef5a8027e0c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:31 UTC) from {username='oryx', principal='oryx/127.20.102.189@KRBTEST.COM'} at 127.0.0.1:51328
I20260925 01:17:31.160627 28227 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d436e1d89e64046928ef5a8027e0c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:31 UTC) from {username='oryx', principal='oryx/127.20.102.189@KRBTEST.COM'} at 127.0.0.1:55004
I20260925 01:17:31.161118 28226 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d436e1d89e64046928ef5a8027e0c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:31 UTC) from {username='oryx', principal='oryx/127.20.102.189@KRBTEST.COM'} at 127.0.0.1:55004
I20260925 01:17:31.161406 28413 tablet_replica.cc:330] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a: stopping tablet replica
I20260925 01:17:31.161388 28086 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d436e1d89e64046928ef5a8027e0c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:31 UTC) from {username='oryx', principal='oryx/127.20.102.189@KRBTEST.COM'} at 127.0.0.1:59190
I20260925 01:17:31.161590 28413 raft_consensus.cc:2229] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:31.161870 28413 raft_consensus.cc:2258] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:31.161881 28085 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d436e1d89e64046928ef5a8027e0c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:31 UTC) from {username='oryx', principal='oryx/127.20.102.189@KRBTEST.COM'} at 127.0.0.1:59190
I20260925 01:17:31.162926 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 27794
I20260925 01:17:31.165539 28413 ts_tablet_manager.cc:1905] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:31.171551 28412 tablet_replica.cc:330] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656: stopping tablet replica
I20260925 01:17:31.171821 28412 raft_consensus.cc:2229] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:31.172191 28412 raft_consensus.cc:2258] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:31.173627 28412 ts_tablet_manager.cc:1905] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:31.174618 28414 tablet_replica.cc:330] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32: stopping tablet replica
I20260925 01:17:31.174817 28414 raft_consensus.cc:2229] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:17:31.174815 28413 ts_tablet_manager.cc:1918] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:31.175048 28413 log.cc:1196] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/0d436e1d89e64046928ef5a8027e0c08
I20260925 01:17:31.175240 28414 raft_consensus.cc:2258] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:31.175736 28413 ts_tablet_manager.cc:1939] T 0d436e1d89e64046928ef5a8027e0c08 P 03178d57cd3d4370aa02d2a3dbcadf1a: Deleting consensus metadata
I20260925 01:17:31.176249 28414 ts_tablet_manager.cc:1905] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:31.177079 27618 catalog_manager.cc:4841] TS 03178d57cd3d4370aa02d2a3dbcadf1a (127.20.102.131:36489): tablet 0d436e1d89e64046928ef5a8027e0c08 (table test-table [id=7fec254bca84410e9ffa690634f65e66]) successfully deleted
W20260925 01:17:31.178407 27618 catalog_manager.cc:4808] TS 03178d57cd3d4370aa02d2a3dbcadf1a (127.20.102.131:36489): delete failed for tablet 0d436e1d89e64046928ef5a8027e0c08 because the tablet was not found. No further retry: Not found: Tablet not found: 0d436e1d89e64046928ef5a8027e0c08
I20260925 01:17:31.181253 28412 ts_tablet_manager.cc:1918] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:31.181416 28412 log.cc:1196] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/0d436e1d89e64046928ef5a8027e0c08
I20260925 01:17:31.181973 28412 ts_tablet_manager.cc:1939] T 0d436e1d89e64046928ef5a8027e0c08 P 6804c301ed464eaaa95340cae0f1b656: Deleting consensus metadata
I20260925 01:17:31.183138 27619 catalog_manager.cc:4841] TS 6804c301ed464eaaa95340cae0f1b656 (127.20.102.129:38049): tablet 0d436e1d89e64046928ef5a8027e0c08 (table test-table [id=7fec254bca84410e9ffa690634f65e66]) successfully deleted
I20260925 01:17:31.183602 28414 ts_tablet_manager.cc:1918] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:31.183746 28414 log.cc:1196] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/0d436e1d89e64046928ef5a8027e0c08
I20260925 01:17:31.184202 28414 ts_tablet_manager.cc:1939] T 0d436e1d89e64046928ef5a8027e0c08 P 2338a1bec88f4c1d89f7b19f487fab32: Deleting consensus metadata
I20260925 01:17:31.185216 27617 catalog_manager.cc:4841] TS 2338a1bec88f4c1d89f7b19f487fab32 (127.20.102.130:43921): tablet 0d436e1d89e64046928ef5a8027e0c08 (table test-table [id=7fec254bca84410e9ffa690634f65e66]) successfully deleted
W20260925 01:17:31.185567 27617 catalog_manager.cc:4808] TS 2338a1bec88f4c1d89f7b19f487fab32 (127.20.102.130:43921): delete failed for tablet 0d436e1d89e64046928ef5a8027e0c08 because the tablet was not found. No further retry: Not found: Tablet not found: 0d436e1d89e64046928ef5a8027e0c08
I20260925 01:17:31.190707 28383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.163939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51342 (local address 127.20.102.129:38049)
0925 01:17:31.164179 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:31.164185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:31.176515 (+ 12330us) server_negotiation.cc:407] Connection header received
0925 01:17:31.176964 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:31.176970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:31.177076 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:31.177151 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:31.179048 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.179807 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.181110 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.181712 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.181861 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:31.185064 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:31.185107 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:31.185127 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:31.185183 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:31.187904 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:31.188569 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:31.188579 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:31.188583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:31.188809 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:31.189185 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:31.189190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:31.189194 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:31.189564 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:31.189862 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:31.190139 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 01:17:31.190370 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:17:31.191299 27998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:31.285780 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 27794
I20260925 01:17:31.304037 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 28015
I20260925 01:17:31.318001 28384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.304864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59216 (local address 127.20.102.130:43921)
0925 01:17:31.305055 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:31.305062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:31.305154 (+ 92us) server_negotiation.cc:407] Connection header received
0925 01:17:31.305364 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:31.305370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:31.305473 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:31.305585 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:31.306883 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.307631 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.308936 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.309344 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.309489 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:31.311940 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:31.311964 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:31.311981 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:31.312013 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:31.315040 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:31.315711 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:31.315722 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:31.315731 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:31.315863 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:31.316273 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:31.316277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:31.316279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:31.316637 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:31.317173 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:31.317444 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 01:17:31.317681 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:17:31.318728 28146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:31.426231 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.425741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56540)
0925 01:17:31.425958 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.426062 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 01:17:31.427423 28385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.427006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37687 (local address 127.0.0.1:56548)
0925 01:17:31.427176 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:31.427282 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:37687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 01:17:31.440809 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 28015
I20260925 01:17:31.464457 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 28164
I20260925 01:17:31.478144 28386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:31.465392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55006 (local address 127.20.102.131:36489)
0925 01:17:31.465597 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:31.465602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:31.465623 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:17:31.465742 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:31.465746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:31.465831 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:31.465924 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:31.467264 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.468009 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.469300 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:31.469798 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:31.469907 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:31.472234 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:31.472252 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:31.472269 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:31.472302 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:31.475486 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:31.476136 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:31.476143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:31.476146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:31.476279 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:31.476630 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:31.476634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:31.476636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:31.477102 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:31.477318 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:31.477615 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 01:17:31.477788 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 01:17:31.478909 28287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:31.572651 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 28164
I20260925 01:17:31.590020 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 27495
I20260925 01:17:31.591256 27663 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:31.716782 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 27495
I20260925 01:17:31.739115 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 27594
I20260925 01:17:31.740204 27852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:31.861639 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 27594
2026-09-25T01:17:31Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9505 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:17:31 dist-test-slave-s12s krb5kdc[28423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:17:31 dist-test-slave-s12s krb5kdc[28423](info): set up 2 sockets
Sep 25 01:17:31 dist-test-slave-s12s krb5kdc[28423](info): commencing operation
krb5kdc: starting...
W20260925 01:17:33.001716 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:17:33 dist-test-slave-s12s krb5kdc[28423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:17:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:17:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.190.keytab.
Entry for principal oryx/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.190.keytab.
I20260925 01:17:33.220196 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:41721
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.190.keytab
--principal=oryx/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:41721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:33.553988 28436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:33.554251 28436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:33.554337 28436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:33.561652 28436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:33.561753 28436 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 01:17:33.561822 28436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:33.561911 28436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:33.561998 28436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:33.562104 28436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:17:33.571684 28436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:41721
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.190.keytab
--principal=oryx/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:41721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:33.572937 28436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:33.574231 28436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:33.585958 28441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:33.586118 28442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:33.586158 28444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:33.587190 28436 server_base.cc:1034] running on GCE node
I20260925 01:17:33.587916 28436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:33.589368 28436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:33.590595 28436 hybrid_clock.cc:648] HybridClock initialized: now 1790299053590527 us; error 39 us; skew 500 ppm
I20260925 01:17:33.590937 28436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:33 dist-test-slave-s12s krb5kdc[28423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299053, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:33.594086 28436 init.cc:377] Logged in from keytab as oryx/127.20.102.190@KRBTEST.COM (short username oryx)
I20260925 01:17:33.595556 28436 webserver.cc:460] Webserver started at http://127.20.102.190:45069/ using document root <none> and password file <none>
I20260925 01:17:33.596259 28436 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:33.596377 28436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:33.596767 28436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:33.599627 28436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "8d3030c7f7ef40efa3636e077d7a77e9"
format_stamp: "Formatted at 2026-09-25 01:17:33 on dist-test-slave-s12s"
server_key: "f0d580028e26537b6c5b339d113cc055"
server_key_iv: "cdd3815fdb02094c4f5ca756456182b9"
server_key_version: "encryptionkey@0"
I20260925 01:17:33.600396 28436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "8d3030c7f7ef40efa3636e077d7a77e9"
format_stamp: "Formatted at 2026-09-25 01:17:33 on dist-test-slave-s12s"
server_key: "f0d580028e26537b6c5b339d113cc055"
server_key_iv: "cdd3815fdb02094c4f5ca756456182b9"
server_key_version: "encryptionkey@0"
I20260925 01:17:33.605398 28436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 01:17:33.609149 28451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:33.610538 28436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 01:17:33.610739 28436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "8d3030c7f7ef40efa3636e077d7a77e9"
format_stamp: "Formatted at 2026-09-25 01:17:33 on dist-test-slave-s12s"
server_key: "f0d580028e26537b6c5b339d113cc055"
server_key_iv: "cdd3815fdb02094c4f5ca756456182b9"
server_key_version: "encryptionkey@0"
I20260925 01:17:33.610978 28436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:33.638486 28436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:33.642707 28436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:33.642988 28436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:33.664444 28436 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:41721
I20260925 01:17:33.664463 28512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:41721 every 8 connection(s)
I20260925 01:17:33.665829 28436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:17:33.669549 28513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:33.671358 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 28436
I20260925 01:17:33.671579 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:17:33.672446 20890 external_mini_cluster.cc:1428] Setting key daffaa28a40c7951467119b73b16ea7f
I20260925 01:17:33.682181 28513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9: Bootstrap starting.
I20260925 01:17:33.686028 28513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9: Neither blocks nor log segments found. Creating new log.
Sep 25 01:17:33 dist-test-slave-s12s krb5kdc[28423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
I20260925 01:17:33.687194 28513 log.cc:824] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:33.690575 28513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9: No bootstrap required, opened a new log
I20260925 01:17:33.696761 28516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:33.674453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.20.102.190:41721)
0925 01:17:33.675294 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:33.675307 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:33.675348 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:17:33.677142 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:33.677163 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:33.677529 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:33.677820 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:33.679559 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:33.680954 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:33.682708 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:33.683343 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:33.683497 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:33.688353 (+ 4856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:33.688361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:33.688374 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:33.688406 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:33.692304 (+ 3898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:33.693049 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:33.693056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:33.693064 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:33.693295 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:33.693681 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:33.693686 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:33.693690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:33.694367 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:33.694661 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:33.694991 (+ 330us) server_negotiation.cc:299] Negotiation successful
0925 01:17:33.695422 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":406,"threads_started":1}
I20260925 01:17:33.700070 28513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3030c7f7ef40efa3636e077d7a77e9" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 41721 } }
I20260925 01:17:33.700376 28513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:33.700443 28513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d3030c7f7ef40efa3636e077d7a77e9, State: Initialized, Role: FOLLOWER
I20260925 01:17:33.700958 28513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3030c7f7ef40efa3636e077d7a77e9" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 41721 } }
I20260925 01:17:33.701164 28513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:17:33.701256 28513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:17:33.701372 28513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:33.703387 28513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3030c7f7ef40efa3636e077d7a77e9" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 41721 } }
I20260925 01:17:33.703809 28513 leader_election.cc:302] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d3030c7f7ef40efa3636e077d7a77e9; no voters:
I20260925 01:17:33.704453 28513 leader_election.cc:288] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:17:33.704640 28518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:33.705574 28518 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [term 1 LEADER]: Becoming Leader. State: Replica: 8d3030c7f7ef40efa3636e077d7a77e9, State: Running, Role: LEADER
I20260925 01:17:33.706202 28518 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3030c7f7ef40efa3636e077d7a77e9" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 41721 } }
I20260925 01:17:33.706614 28513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:33.709249 28519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d3030c7f7ef40efa3636e077d7a77e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3030c7f7ef40efa3636e077d7a77e9" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 41721 } } }
I20260925 01:17:33.709658 28519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:33.710122 28520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d3030c7f7ef40efa3636e077d7a77e9. Latest consensus state: current_term: 1 leader_uuid: "8d3030c7f7ef40efa3636e077d7a77e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d3030c7f7ef40efa3636e077d7a77e9" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 41721 } } }
I20260925 01:17:33.710496 28520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:33.716863 28526 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:17:33.721082 28526 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:17:33.730075 28526 catalog_manager.cc:1347] Generated new cluster ID: edb98e7021a3411ca75b4bc7c4a3ccad
I20260925 01:17:33.730186 28526 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:17:33.743281 28526 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:17:33.745276 28526 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:17:33.758658 28526 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9: Generated new TSK 0
I20260925 01:17:33.759709 28526 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.129.keytab.
Entry for principal oryx/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.129.keytab.
I20260925 01:17:33.820483 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:41721
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.129.keytab
--principal=oryx/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:34.145596 28540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:34.145885 28540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:34.146113 28540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:34.153479 28540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:34.153579 28540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:34.153759 28540 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:17:34.153884 28540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:17:34.164660 28540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.129.keytab
--principal=oryx/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.102.190:41721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:34.165915 28540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:34.167296 28540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:34.180275 28545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:34.180365 28548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:34.180335 28546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:34.181183 28540 server_base.cc:1034] running on GCE node
I20260925 01:17:34.181766 28540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:34.182636 28540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:34.183884 28540 hybrid_clock.cc:648] HybridClock initialized: now 1790299054183843 us; error 54 us; skew 500 ppm
I20260925 01:17:34.184175 28540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:34 dist-test-slave-s12s krb5kdc[28423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299054, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:34.187407 28540 init.cc:377] Logged in from keytab as oryx/127.20.102.129@KRBTEST.COM (short username oryx)
I20260925 01:17:34.188776 28540 webserver.cc:460] Webserver started at http://127.20.102.129:36081/ using document root <none> and password file <none>
I20260925 01:17:34.189425 28540 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:34.189512 28540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:34.189827 28540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:34.192559 28540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "c42f998c7c4947259fd79fdfd9ff7644"
format_stamp: "Formatted at 2026-09-25 01:17:34 on dist-test-slave-s12s"
server_key: "98d02795d62412a9c54d5dae4026e0e5"
server_key_iv: "f16b7d1b0e78a729eec74749c09b048d"
server_key_version: "encryptionkey@0"
I20260925 01:17:34.193365 28540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "c42f998c7c4947259fd79fdfd9ff7644"
format_stamp: "Formatted at 2026-09-25 01:17:34 on dist-test-slave-s12s"
server_key: "98d02795d62412a9c54d5dae4026e0e5"
server_key_iv: "f16b7d1b0e78a729eec74749c09b048d"
server_key_version: "encryptionkey@0"
I20260925 01:17:34.198362 28540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 01:17:34.201758 28555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:34.203234 28540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 01:17:34.203445 28540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "c42f998c7c4947259fd79fdfd9ff7644"
format_stamp: "Formatted at 2026-09-25 01:17:34 on dist-test-slave-s12s"
server_key: "98d02795d62412a9c54d5dae4026e0e5"
server_key_iv: "f16b7d1b0e78a729eec74749c09b048d"
server_key_version: "encryptionkey@0"
I20260925 01:17:34.203718 28540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:34.249162 28540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:34.253150 28540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:34.253378 28540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:34.256053 28540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:34.256150 28540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:34.256356 28540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:34.256440 28540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:17:34 dist-test-slave-s12s krb5kdc[28423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299054, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
I20260925 01:17:34.278594 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:34.256615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:40773 (local address 127.20.102.190:41721)
0925 01:17:34.257464 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:34.257472 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:34.258469 (+ 997us) server_negotiation.cc:407] Connection header received
0925 01:17:34.260755 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:34.260764 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:34.260846 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:34.260972 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:34.263128 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.264172 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.266784 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.267380 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.267519 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:34.271622 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:34.271629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:34.271639 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:34.271673 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:34.274439 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:34.275528 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:34.275534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:34.275537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:34.275715 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:34.276386 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:34.276394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:34.276400 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:34.276743 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:34.276914 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:34.278055 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0925 01:17:34.278324 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":593,"threads_started":1}
I20260925 01:17:34.280387 28565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:34.257378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:41721 (local address 127.20.102.129:40773)
0925 01:17:34.258252 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:34.258311 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:34.260337 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:34.261269 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:34.261283 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:34.262009 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:34.262919 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:34.262949 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.264329 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.264335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:34.266632 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:34.266644 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.267648 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.267656 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:34.267862 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:34.268568 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:34.268589 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:34.271396 (+ 2807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:34.274591 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:34.274603 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:34.274626 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:34.275390 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:34.275864 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:34.275873 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:34.275880 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:34.276228 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:34.276889 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:34.276900 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:34.277480 (+ 580us) client_negotiation.cc:769] Sending connection context
0925 01:17:34.277883 (+ 403us) client_negotiation.cc:240] Negotiation successful
0925 01:17:34.278602 (+ 719us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":380,"threads_started":1}
I20260925 01:17:34.311393 28540 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:36191
I20260925 01:17:34.311452 28675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:36191 every 8 connection(s)
I20260925 01:17:34.313118 28540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:17:34.315469 28676 heartbeater.cc:347] Connected to a master server at 127.20.102.190:41721
I20260925 01:17:34.315739 28676 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:34.316641 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:34.299672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:58831 (local address 127.20.102.190:41721)
0925 01:17:34.299877 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:34.299885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:34.300909 (+ 1024us) server_negotiation.cc:407] Connection header received
0925 01:17:34.301194 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:34.301200 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:34.301276 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:34.301383 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:34.302314 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.303324 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.305323 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.307508 (+ 2185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.307680 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:34.309878 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:34.309887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:34.309895 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:34.309935 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:34.313595 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:34.314566 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:34.314572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:34.314576 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:34.314721 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:34.315269 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:34.315274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:34.315277 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:34.315651 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:34.315782 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:34.316246 (+ 464us) server_negotiation.cc:299] Negotiation successful
0925 01:17:34.316406 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:17:34.316609 28676 heartbeater.cc:511] Master 127.20.102.190:41721 requested a full tablet report, sending...
I20260925 01:17:34.317039 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:34.299859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:41721 (local address 127.20.102.129:58831)
0925 01:17:34.300774 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:34.300798 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:34.300973 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:34.301414 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:34.301419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:34.301881 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:34.302163 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:34.302173 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.303476 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.303481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:34.305122 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:34.305137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.307739 (+ 2602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.307746 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:34.307946 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:34.308662 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:34.308731 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:34.309683 (+ 952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:34.313772 (+ 4089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:34.313802 (+ 30us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:34.313806 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:34.314377 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:34.314867 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:34.314872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:34.314875 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:34.315115 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:34.315775 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:34.315781 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:34.315990 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 01:17:34.316164 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 01:17:34.316516 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":556,"threads_started":1}
I20260925 01:17:34.318790 28467 ts_manager.cc:194] Registered new tserver with Master: c42f998c7c4947259fd79fdfd9ff7644 (127.20.102.129:36191)
I20260925 01:17:34.320758 28467 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:40773
I20260925 01:17:34.322016 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 28540
I20260925 01:17:34.322212 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:17:34.322865 20890 external_mini_cluster.cc:1428] Setting key b2fa0dbffc0e3883ef6777846a0ccacf
WARNING: no policy specified for oryx/127.20.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.130.keytab.
Entry for principal oryx/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.130.keytab.
I20260925 01:17:34.367239 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:41721
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.130.keytab
--principal=oryx/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:34.696635 28680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:34.696955 28680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:34.697054 28680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:34.704173 28680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:34.704268 28680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:34.704349 28680 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:17:34.704432 28680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:17:34.714375 28680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.130.keytab
--principal=oryx/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.102.190:41721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:34.715660 28680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:34.716977 28680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:34.730413 28685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:34.730660 28688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:34.730649 28686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:34.731837 28680 server_base.cc:1034] running on GCE node
I20260925 01:17:34.732474 28680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:34.733413 28680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:34.734673 28680 hybrid_clock.cc:648] HybridClock initialized: now 1790299054734596 us; error 49 us; skew 500 ppm
I20260925 01:17:34.734959 28680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:34 dist-test-slave-s12s krb5kdc[28423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299054, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:34.738030 28680 init.cc:377] Logged in from keytab as oryx/127.20.102.130@KRBTEST.COM (short username oryx)
I20260925 01:17:34.739336 28680 webserver.cc:460] Webserver started at http://127.20.102.130:43929/ using document root <none> and password file <none>
I20260925 01:17:34.740024 28680 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:34.740151 28680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:34.740475 28680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:34.743273 28680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef"
format_stamp: "Formatted at 2026-09-25 01:17:34 on dist-test-slave-s12s"
server_key: "8cc156db2ae97a926f97121cd25ac749"
server_key_iv: "2ca74583627b60e8eef6cfb97c795774"
server_key_version: "encryptionkey@0"
I20260925 01:17:34.744077 28680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef"
format_stamp: "Formatted at 2026-09-25 01:17:34 on dist-test-slave-s12s"
server_key: "8cc156db2ae97a926f97121cd25ac749"
server_key_iv: "2ca74583627b60e8eef6cfb97c795774"
server_key_version: "encryptionkey@0"
I20260925 01:17:34.748925 28680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 01:17:34.752169 28695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:34.753448 28680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:34.753619 28680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef"
format_stamp: "Formatted at 2026-09-25 01:17:34 on dist-test-slave-s12s"
server_key: "8cc156db2ae97a926f97121cd25ac749"
server_key_iv: "2ca74583627b60e8eef6cfb97c795774"
server_key_version: "encryptionkey@0"
I20260925 01:17:34.753834 28680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:34.767907 28680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:34.771955 28680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:34.772238 28680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:34.774833 28680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:34.774948 28680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:34.775100 28680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:34.775194 28680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:17:34 dist-test-slave-s12s krb5kdc[28423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299054, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
I20260925 01:17:34.799930 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:34.775701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:36143 (local address 127.20.102.190:41721)
0925 01:17:34.775901 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:34.775907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:34.777452 (+ 1545us) server_negotiation.cc:407] Connection header received
0925 01:17:34.780104 (+ 2652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:34.780111 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:34.780208 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:34.780344 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:34.782802 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.784127 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.786661 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.787459 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.787621 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:34.792536 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:34.792545 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:34.792550 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:34.792604 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:34.795969 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:34.796999 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:34.797007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:34.797014 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:34.797192 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:34.797837 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:34.797844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:34.797850 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:34.798193 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:34.798379 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:34.799336 (+ 957us) server_negotiation.cc:299] Negotiation successful
0925 01:17:34.799568 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 01:17:34.801614 28704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:34.776349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:41721 (local address 127.20.102.130:36143)
0925 01:17:34.777255 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:34.777304 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:34.779693 (+ 2389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:34.780659 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:34.780717 (+ 58us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:34.781575 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:34.782561 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:34.782593 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.784306 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.784310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:34.786426 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:34.786439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.787810 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.787819 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:34.788006 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:34.788795 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:34.788816 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:34.792245 (+ 3429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:34.796131 (+ 3886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:34.796145 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:34.796166 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:34.796841 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:34.797358 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:34.797370 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:34.797376 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:34.797692 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:34.798345 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:34.798353 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:34.798793 (+ 440us) client_negotiation.cc:769] Sending connection context
0925 01:17:34.799212 (+ 419us) client_negotiation.cc:240] Negotiation successful
0925 01:17:34.799790 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":436,"threads_started":1}
I20260925 01:17:34.825496 28680 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:44795
I20260925 01:17:34.825584 28814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:44795 every 8 connection(s)
I20260925 01:17:34.826844 28680 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:17:34.829484 28815 heartbeater.cc:347] Connected to a master server at 127.20.102.190:41721
I20260925 01:17:34.829741 28815 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:34.830556 28815 heartbeater.cc:511] Master 127.20.102.190:41721 requested a full tablet report, sending...
I20260925 01:17:34.832038 28467 ts_manager.cc:194] Registered new tserver with Master: 39928671a1cf4d6dbdc6a4e7ff1071ef (127.20.102.130:44795)
I20260925 01:17:34.832743 28780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:34.809610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:41721 (local address 127.20.102.130:44931)
0925 01:17:34.813564 (+ 3954us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:34.813594 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:34.813746 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:34.814110 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:34.814115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:34.814871 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:34.815409 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:34.815430 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.817001 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.817007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:34.818578 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:34.818598 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.819598 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.819605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:34.819801 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:34.825325 (+ 5524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:34.825358 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:34.826283 (+ 925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:34.830177 (+ 3894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:34.830185 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:34.830188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:34.830597 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:34.831023 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:34.831028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:34.831032 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:34.831170 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:34.831800 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:34.831804 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:34.831955 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 01:17:34.832098 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 01:17:34.832392 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3757,"thread_start_us":3480,"threads_started":1}
I20260925 01:17:34.832790 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:34.809987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:44931 (local address 127.20.102.190:41721)
0925 01:17:34.810152 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:34.810156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:34.813670 (+ 3514us) server_negotiation.cc:407] Connection header received
0925 01:17:34.813891 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:34.813896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:34.813965 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:34.814085 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:34.815633 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.816863 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.818789 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:34.819375 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:34.819535 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:34.826549 (+ 7014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:34.826559 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:34.826564 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:34.826610 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:34.830041 (+ 3431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:34.830770 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:34.830774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:34.830776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:34.830903 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:34.831286 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:34.831292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:34.831296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:34.831676 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:34.831828 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:34.832223 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 01:17:34.832435 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:17:34.833248 28467 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:36143
I20260925 01:17:34.837039 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 28680
I20260925 01:17:34.837224 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:17:34.837671 20890 external_mini_cluster.cc:1428] Setting key a6eb7cf100c350b845bd3836f870ed63
WARNING: no policy specified for oryx/127.20.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.131.keytab.
Entry for principal oryx/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.131.keytab.
I20260925 01:17:34.884241 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:41721
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.131.keytab
--principal=oryx/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:35.227463 28819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:35.227849 28819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:35.227964 28819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:35.235522 28819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:35.235621 28819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:35.235706 28819 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 01:17:35.235786 28819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:17:35.245577 28819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.131.keytab
--principal=oryx/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.20.102.190:41721
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:35.246814 28819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:35.248065 28819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:35.261304 28824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:35.261269 28825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:35.261340 28827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:35.262188 28819 server_base.cc:1034] running on GCE node
I20260925 01:17:35.262879 28819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:35.263830 28819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:35.265055 28819 hybrid_clock.cc:648] HybridClock initialized: now 1790299055264974 us; error 57 us; skew 500 ppm
I20260925 01:17:35.265354 28819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:35 dist-test-slave-s12s krb5kdc[28423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299055, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:35.268440 28819 init.cc:377] Logged in from keytab as oryx/127.20.102.131@KRBTEST.COM (short username oryx)
I20260925 01:17:35.270015 28819 webserver.cc:460] Webserver started at http://127.20.102.131:37897/ using document root <none> and password file <none>
I20260925 01:17:35.270715 28819 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:35.270867 28819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:35.271199 28819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:35.274111 28819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "b86efccdf27348cb8230e37b637d9e70"
format_stamp: "Formatted at 2026-09-25 01:17:35 on dist-test-slave-s12s"
server_key: "c2cc08bf801886e36f59ff0c3e4e8dab"
server_key_iv: "f66d655f362310c95713cbf56dc28dba"
server_key_version: "encryptionkey@0"
I20260925 01:17:35.274902 28819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "b86efccdf27348cb8230e37b637d9e70"
format_stamp: "Formatted at 2026-09-25 01:17:35 on dist-test-slave-s12s"
server_key: "c2cc08bf801886e36f59ff0c3e4e8dab"
server_key_iv: "f66d655f362310c95713cbf56dc28dba"
server_key_version: "encryptionkey@0"
I20260925 01:17:35.279907 28819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 01:17:35.283314 28834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:35.284623 28819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:17:35.284833 28819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "b86efccdf27348cb8230e37b637d9e70"
format_stamp: "Formatted at 2026-09-25 01:17:35 on dist-test-slave-s12s"
server_key: "c2cc08bf801886e36f59ff0c3e4e8dab"
server_key_iv: "f66d655f362310c95713cbf56dc28dba"
server_key_version: "encryptionkey@0"
I20260925 01:17:35.285068 28819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:35.309504 28819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:35.314392 28819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:35.314746 28819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:35.317813 28819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:35.317914 28819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:35.318030 28819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:35.318109 28819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:35.323485 28676 heartbeater.cc:503] Master 127.20.102.190:41721 was elected leader, sending a full tablet report...
Sep 25 01:17:35 dist-test-slave-s12s krb5kdc[28423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299055, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
I20260925 01:17:35.343293 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.318425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:54757 (local address 127.20.102.190:41721)
0925 01:17:35.318605 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:35.318610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:35.320065 (+ 1455us) server_negotiation.cc:407] Connection header received
0925 01:17:35.322261 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:35.322267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:35.322358 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:35.322471 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:35.325452 (+ 2981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.326770 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.329399 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.330000 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.330137 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.335840 (+ 5703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:35.335846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:35.335856 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:35.335888 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:35.339249 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:35.340254 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:35.340262 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:35.340267 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:35.340427 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:35.341088 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:35.341094 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:35.341097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:35.341511 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:35.341663 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:35.342685 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0925 01:17:35.342919 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 01:17:35.344323 28843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.319091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:41721 (local address 127.20.102.131:54757)
0925 01:17:35.319899 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:35.319935 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:35.321936 (+ 2001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:35.322773 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:35.322787 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:35.323571 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:35.325203 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:35.325236 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.326955 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.326962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:35.329236 (+ 2274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:35.329256 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.330263 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.330276 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:35.330636 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.331543 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:35.331581 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:35.335609 (+ 4028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:35.339401 (+ 3792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:35.339410 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:35.339430 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:35.340074 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:35.340565 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:35.340570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:35.340574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:35.340955 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:35.341729 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:35.341735 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:35.342236 (+ 501us) client_negotiation.cc:769] Sending connection context
0925 01:17:35.342558 (+ 322us) client_negotiation.cc:240] Negotiation successful
0925 01:17:35.343070 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":375,"threads_started":1}
I20260925 01:17:35.365464 28819 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:36847
I20260925 01:17:35.365484 28953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:36847 every 8 connection(s)
I20260925 01:17:35.366753 28819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:17:35.369153 28954 heartbeater.cc:347] Connected to a master server at 127.20.102.190:41721
I20260925 01:17:35.369381 28954 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:35.370039 28954 heartbeater.cc:511] Master 127.20.102.190:41721 requested a full tablet report, sending...
I20260925 01:17:35.370814 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.355781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:34847 (local address 127.20.102.190:41721)
0925 01:17:35.355942 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:35.355946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:35.356629 (+ 683us) server_negotiation.cc:407] Connection header received
0925 01:17:35.356946 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:35.356956 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:35.357058 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:35.357177 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:35.359040 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.359912 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.362108 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.362531 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.362665 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.364859 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:35.364866 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:35.364871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:35.364908 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:35.367792 (+ 2884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:35.368673 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:35.368707 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:35.368713 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:35.368874 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:35.369353 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:35.369357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:35.369359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:35.369671 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:35.369827 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:35.370251 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 01:17:35.370462 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:17:35.370949 28929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.355844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:41721 (local address 127.20.102.131:34847)
0925 01:17:35.356479 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:35.356504 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:35.356762 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:35.357384 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:35.357390 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:35.358121 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:35.358660 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:35.358677 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.360072 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.360077 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:35.361943 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:35.361956 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.362789 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.362798 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:35.362945 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.363613 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:35.363630 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:35.364620 (+ 990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:35.367970 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:35.367975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:35.367979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:35.368550 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:35.369043 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:35.369048 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:35.369051 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:35.369194 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:35.369791 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:35.369796 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:35.369997 (+ 201us) client_negotiation.cc:769] Sending connection context
0925 01:17:35.370138 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 01:17:35.370449 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":342,"threads_started":1}
I20260925 01:17:35.371565 28468 ts_manager.cc:194] Registered new tserver with Master: b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847)
I20260925 01:17:35.372494 28468 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:54757
I20260925 01:17:35.375314 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 28819
I20260925 01:17:35.375501 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:17:35.375957 20890 external_mini_cluster.cc:1428] Setting key e8e62295aa32acc94573d5261464a781
I20260925 01:17:35.378782 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 01:17:35 dist-test-slave-s12s krb5kdc[28423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299055, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 01:17:35 dist-test-slave-s12s krb5kdc[28423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:35 dist-test-slave-s12s krb5kdc[28423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:17:35.428141 28564 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:35.417928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43402 (local address 127.20.102.190:41721)
0925 01:17:35.418209 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:35.418215 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:35.418367 (+ 152us) server_negotiation.cc:407] Connection header received
0925 01:17:35.418517 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:35.418521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:35.418588 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:35.418675 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:35.419947 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.420663 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.421916 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.422314 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.422412 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.427818 (+ 5406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43402: BlockingRecv error: recv got EOF from 127.0.0.1:43402 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 25 01:17:35 dist-test-slave-s12s krb5kdc[28423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299055, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
I20260925 01:17:35.443636 28564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.430930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43412 (local address 127.20.102.190:41721)
0925 01:17:35.431108 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:35.431112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:35.431386 (+ 274us) server_negotiation.cc:407] Connection header received
0925 01:17:35.431523 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:35.431526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:35.431598 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:35.431680 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:35.432808 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.433548 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.434724 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.435162 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.435283 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.438893 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:35.438898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:35.438902 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:35.438934 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:35.441669 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:35.442215 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:35.442219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:35.442221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:35.442332 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:35.442627 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:35.442630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:35.442632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:35.442864 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:35.443098 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:35.443255 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 01:17:35.443422 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:17:35.447686 28468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:17:35.454317 28468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:17:35.477324 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.466524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:36847 (local address 127.0.0.1:41724)
0925 01:17:35.467749 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:35.467799 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:35.468000 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:35.469275 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:35.469279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:35.469303 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:35.469806 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:35.469816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.472116 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.472122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:35.474274 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:35.474286 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.476070 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.476079 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:35.476373 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.476446 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 01:17:35.476578 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 01:17:35.476786 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1100,"mutex_wait_us":22,"spinlock_wait_cycles":2688,"thread_start_us":1023,"threads_started":1}
I20260925 01:17:35.477528 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.466467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46266 (local address 127.20.102.130:44795)
0925 01:17:35.467139 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:35.467148 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:35.468189 (+ 1041us) server_negotiation.cc:407] Connection header received
0925 01:17:35.468308 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:35.468318 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:35.468596 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:35.468815 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:35.469886 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.471409 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.474217 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.475635 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.475806 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.476013 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:35.477072 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0925 01:17:35.477244 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":395,"threads_started":1}
I20260925 01:17:35.478215 28974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.466238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:44795 (local address 127.0.0.1:46266)
0925 01:17:35.467749 (+ 1511us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:35.467799 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:35.467990 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:35.469051 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:35.469058 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:35.469099 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:35.469607 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:35.469622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.471574 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.471579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:35.474032 (+ 2453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:35.474047 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.476153 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.476158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:35.476777 (+ 619us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.476842 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 01:17:35.477745 (+ 903us) client_negotiation.cc:240] Negotiation successful
0925 01:17:35.477878 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"mutex_wait_us":385,"spinlock_wait_cycles":6528,"thread_start_us":1279,"threads_started":1}
I20260925 01:17:35.478449 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.467280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41724 (local address 127.20.102.131:36847)
0925 01:17:35.468650 (+ 1370us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:35.468658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:35.468718 (+ 60us) server_negotiation.cc:407] Connection header received
0925 01:17:35.468811 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:35.468817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:35.469058 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:35.469984 (+ 926us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:35.470353 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.471933 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.474460 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.475814 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.477594 (+ 1780us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.477869 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:35.477943 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 01:17:35.478116 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1197,"thread_start_us":1014,"threads_started":1}
I20260925 01:17:35.481901 28745 tablet_service.cc:1467] Processing CreateTablet for tablet d544e1893e27454a8b482fde7af44102 (DEFAULT_TABLE table=test-table [id=0c0b291bac95445d828c0eb45bcd9a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:35.482050 28884 tablet_service.cc:1467] Processing CreateTablet for tablet d544e1893e27454a8b482fde7af44102 (DEFAULT_TABLE table=test-table [id=0c0b291bac95445d828c0eb45bcd9a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:35.483299 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d544e1893e27454a8b482fde7af44102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:35.483419 28884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d544e1893e27454a8b482fde7af44102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:35.485631 28976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.468480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:36191 (local address 127.0.0.1:51324)
0925 01:17:35.477271 (+ 8791us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:35.477292 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:35.477505 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:35.478362 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:35.478367 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:35.478393 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:35.478732 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:35.478743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.480781 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.480786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:35.482644 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:35.482659 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.484842 (+ 2183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.484848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:35.485020 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.485065 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 01:17:35.485169 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 01:17:35.485328 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8659,"thread_start_us":8485,"threads_started":1}
I20260925 01:17:35.485679 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.468973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.20.102.129:36191)
0925 01:17:35.476785 (+ 7812us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:35.476794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:35.477389 (+ 595us) server_negotiation.cc:407] Connection header received
0925 01:17:35.477718 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:35.477728 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:35.477958 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:35.478114 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:35.478942 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.480619 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.482789 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.484593 (+ 1804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.484828 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.485007 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:35.485229 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 01:17:35.485405 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7625,"thread_start_us":7454,"threads_started":1}
I20260925 01:17:35.489740 28606 tablet_service.cc:1467] Processing CreateTablet for tablet d544e1893e27454a8b482fde7af44102 (DEFAULT_TABLE table=test-table [id=0c0b291bac95445d828c0eb45bcd9a62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:35.491191 28606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d544e1893e27454a8b482fde7af44102. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:35.496443 28978 tablet_bootstrap.cc:492] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Bootstrap starting.
I20260925 01:17:35.498399 28979 tablet_bootstrap.cc:492] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70: Bootstrap starting.
I20260925 01:17:35.501001 28978 tablet_bootstrap.cc:654] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:35.501253 28979 tablet_bootstrap.cc:654] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:35.502193 28979 log.cc:824] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:35.502221 28978 log.cc:824] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:35.505237 28978 tablet_bootstrap.cc:492] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef: No bootstrap required, opened a new log
I20260925 01:17:35.505517 28978 ts_tablet_manager.cc:1397] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260925 01:17:35.505575 28982 tablet_bootstrap.cc:492] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644: Bootstrap starting.
I20260925 01:17:35.506016 28979 tablet_bootstrap.cc:492] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70: No bootstrap required, opened a new log
I20260925 01:17:35.506258 28979 ts_tablet_manager.cc:1397] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260925 01:17:35.508975 28982 tablet_bootstrap.cc:654] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:35.509936 28982 log.cc:824] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:35.512341 28982 tablet_bootstrap.cc:492] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644: No bootstrap required, opened a new log
I20260925 01:17:35.512609 28982 ts_tablet_manager.cc:1397] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 01:17:35.516878 28979 raft_consensus.cc:348] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } }
I20260925 01:17:35.516870 28978 raft_consensus.cc:348] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } }
I20260925 01:17:35.517316 28978 raft_consensus.cc:374] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:35.517315 28979 raft_consensus.cc:374] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:35.517432 28979 raft_consensus.cc:729] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b86efccdf27348cb8230e37b637d9e70, State: Initialized, Role: FOLLOWER
I20260925 01:17:35.517462 28978 raft_consensus.cc:729] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39928671a1cf4d6dbdc6a4e7ff1071ef, State: Initialized, Role: FOLLOWER
I20260925 01:17:35.518072 28979 consensus_queue.cc:260] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } }
I20260925 01:17:35.518072 28978 consensus_queue.cc:260] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } }
I20260925 01:17:35.519184 28954 heartbeater.cc:503] Master 127.20.102.190:41721 was elected leader, sending a full tablet report...
I20260925 01:17:35.520147 28979 ts_tablet_manager.cc:1428] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20260925 01:17:35.521230 28815 heartbeater.cc:503] Master 127.20.102.190:41721 was elected leader, sending a full tablet report...
I20260925 01:17:35.521832 28978 ts_tablet_manager.cc:1428] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Time spent starting tablet: real 0.016s user 0.010s sys 0.005s
I20260925 01:17:35.521677 28982 raft_consensus.cc:348] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } }
I20260925 01:17:35.522078 28982 raft_consensus.cc:374] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:35.522235 28982 raft_consensus.cc:729] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42f998c7c4947259fd79fdfd9ff7644, State: Initialized, Role: FOLLOWER
I20260925 01:17:35.522876 28982 consensus_queue.cc:260] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } }
I20260925 01:17:35.525676 28982 ts_tablet_manager.cc:1428] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260925 01:17:35.565879 28677 tablet.cc:2369] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:17:35.579332 28816 tablet.cc:2369] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:17:35.616366 28986 raft_consensus.cc:482] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:17:35.616745 28986 raft_consensus.cc:504] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } }
I20260925 01:17:35.618551 28986 leader_election.cc:288] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39928671a1cf4d6dbdc6a4e7ff1071ef (127.20.102.130:44795), b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847)
W20260925 01:17:35.618577 28955 tablet.cc:2369] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:17:35.626470 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.620535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:40465 (local address 127.20.102.130:44795)
0925 01:17:35.620775 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:35.620779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:35.620795 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:17:35.620877 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:35.620881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:35.620955 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:35.621084 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:35.621703 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.622711 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.624788 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.625501 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.625704 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.625783 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:35.626067 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 01:17:35.626192 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 01:17:35.626523 28975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.619899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:35809 (local address 127.20.102.131:36847)
0925 01:17:35.620117 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:35.620125 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:35.620366 (+ 241us) server_negotiation.cc:407] Connection header received
0925 01:17:35.620653 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:35.620661 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:35.620769 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:35.620864 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:35.621523 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.622594 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.624832 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.625575 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.625770 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.625843 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:35.626126 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 01:17:35.626263 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:17:35.626673 28987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.618710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:44795 (local address 127.20.102.129:40465)
0925 01:17:35.620080 (+ 1370us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:35.620108 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:35.620279 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:35.621115 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:35.621120 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:35.621145 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:35.621554 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:35.621564 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.622871 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.622880 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:35.624593 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:35.624608 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.625741 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.625747 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:35.625904 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.625930 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 01:17:35.626040 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 01:17:35.626168 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"mutex_wait_us":80,"thread_start_us":954,"threads_started":1}
I20260925 01:17:35.626844 28988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.619341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:36847 (local address 127.20.102.129:35809)
0925 01:17:35.620265 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:35.620284 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:35.620477 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:35.620898 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:35.620902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:35.620926 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:35.621362 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:35.621374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.622745 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.622751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:35.624618 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:35.624631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.625799 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.625805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:35.625959 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.625981 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 01:17:35.626089 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 01:17:35.626207 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":5,"thread_start_us":752,"threads_started":1}
I20260925 01:17:35.627847 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d544e1893e27454a8b482fde7af44102" candidate_uuid: "c42f998c7c4947259fd79fdfd9ff7644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" is_pre_election: true
I20260925 01:17:35.627928 28904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d544e1893e27454a8b482fde7af44102" candidate_uuid: "c42f998c7c4947259fd79fdfd9ff7644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b86efccdf27348cb8230e37b637d9e70" is_pre_election: true
I20260925 01:17:35.628525 28765 raft_consensus.cc:2454] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c42f998c7c4947259fd79fdfd9ff7644 in term 0.
I20260925 01:17:35.628609 28904 raft_consensus.cc:2454] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c42f998c7c4947259fd79fdfd9ff7644 in term 0.
I20260925 01:17:35.629348 28558 leader_election.cc:302] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39928671a1cf4d6dbdc6a4e7ff1071ef, c42f998c7c4947259fd79fdfd9ff7644; no voters:
I20260925 01:17:35.629729 28986 raft_consensus.cc:2781] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:17:35.629889 28986 raft_consensus.cc:482] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:17:35.629994 28986 raft_consensus.cc:3037] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:35.632138 28986 raft_consensus.cc:504] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } }
I20260925 01:17:35.632735 28986 leader_election.cc:288] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [CANDIDATE]: Term 1 election: Requested vote from peers 39928671a1cf4d6dbdc6a4e7ff1071ef (127.20.102.130:44795), b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847)
I20260925 01:17:35.633095 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d544e1893e27454a8b482fde7af44102" candidate_uuid: "c42f998c7c4947259fd79fdfd9ff7644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef"
I20260925 01:17:35.633219 28904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d544e1893e27454a8b482fde7af44102" candidate_uuid: "c42f998c7c4947259fd79fdfd9ff7644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b86efccdf27348cb8230e37b637d9e70"
I20260925 01:17:35.633299 28765 raft_consensus.cc:3037] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:35.633409 28904 raft_consensus.cc:3037] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:35.636535 28904 raft_consensus.cc:2454] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c42f998c7c4947259fd79fdfd9ff7644 in term 1.
I20260925 01:17:35.636535 28765 raft_consensus.cc:2454] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c42f998c7c4947259fd79fdfd9ff7644 in term 1.
I20260925 01:17:35.636965 28558 leader_election.cc:302] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 39928671a1cf4d6dbdc6a4e7ff1071ef, b86efccdf27348cb8230e37b637d9e70, c42f998c7c4947259fd79fdfd9ff7644; no voters:
I20260925 01:17:35.637293 28986 raft_consensus.cc:2781] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:35.637786 28986 raft_consensus.cc:686] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 LEADER]: Becoming Leader. State: Replica: c42f998c7c4947259fd79fdfd9ff7644, State: Running, Role: LEADER
I20260925 01:17:35.638259 28986 consensus_queue.cc:237] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } }
I20260925 01:17:35.642998 28468 catalog_manager.cc:5495] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 reported cstate change: term changed from 0 to 1, leader changed from <none> to c42f998c7c4947259fd79fdfd9ff7644 (127.20.102.129). New cstate: current_term: 1 leader_uuid: "c42f998c7c4947259fd79fdfd9ff7644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } health_report { overall_health: HEALTHY } } }
I20260925 01:17:35.678753 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:35.673089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51332 (local address 127.20.102.129:36191)
0925 01:17:35.673271 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:35.673275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:35.673292 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:17:35.673477 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:35.673481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:35.673558 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:35.673678 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:17:35.674068 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.674739 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.676632 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:35.677077 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:35.677173 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:35.677663 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:17:35.678178 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:17:35.678282 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:35.678422 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 01:17:35.678486 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 01:17:35.688081 28904 raft_consensus.cc:1261] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Refusing update from remote peer c42f998c7c4947259fd79fdfd9ff7644: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:35.688187 28765 raft_consensus.cc:1261] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Refusing update from remote peer c42f998c7c4947259fd79fdfd9ff7644: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:35.689131 28989 consensus_queue.cc:1035] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:35.689497 28986 consensus_queue.cc:1035] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:35.704437 28994 mvcc.cc:205] Tried to move back new op lower bound from 7333064932088262656 to 7333064931897610240. Current Snapshot: MvccSnapshot[applied={T|T < 7333064932088262656}]
I20260925 01:17:35.711674 28993 mvcc.cc:205] Tried to move back new op lower bound from 7333064932088262656 to 7333064931897610240. Current Snapshot: MvccSnapshot[applied={T|T < 7333064932088262656}]
I20260925 01:17:35.714484 28467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:35.714771 28467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:35.718622 28467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9: Sending DeleteTablet for 3 replicas of tablet d544e1893e27454a8b482fde7af44102
I20260925 01:17:35.719726 28745 tablet_service.cc:1514] Processing DeleteTablet for tablet d544e1893e27454a8b482fde7af44102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:35 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:46266
I20260925 01:17:35.720057 28884 tablet_service.cc:1514] Processing DeleteTablet for tablet d544e1893e27454a8b482fde7af44102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:35 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:41724
I20260925 01:17:35.720726 29002 tablet_replica.cc:330] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef: stopping tablet replica
I20260925 01:17:35.720649 28606 tablet_service.cc:1514] Processing DeleteTablet for tablet d544e1893e27454a8b482fde7af44102 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:35 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:51324
I20260925 01:17:35.721304 29002 raft_consensus.cc:2229] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:35.721864 29002 raft_consensus.cc:2258] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:35.723138 28467 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:17:35.723825 28467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:17:35.724721 29002 ts_tablet_manager.cc:1905] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:35.732487 28883 tablet_service.cc:1467] Processing CreateTablet for tablet b18fb525397f421492791136f849094d (DEFAULT_TABLE table=test-table [id=e0e5d2f9236142d5bdcfb0ae9d5bf41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:35.732998 28605 tablet_service.cc:1467] Processing CreateTablet for tablet b18fb525397f421492791136f849094d (DEFAULT_TABLE table=test-table [id=e0e5d2f9236142d5bdcfb0ae9d5bf41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:35.733150 28883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18fb525397f421492791136f849094d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:35.733577 28605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18fb525397f421492791136f849094d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:35.736984 28745 tablet_service.cc:1467] Processing CreateTablet for tablet b18fb525397f421492791136f849094d (DEFAULT_TABLE table=test-table [id=e0e5d2f9236142d5bdcfb0ae9d5bf41a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:35.737396 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b18fb525397f421492791136f849094d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:35.740078 28979 tablet_bootstrap.cc:492] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70: Bootstrap starting.
I20260925 01:17:35.741748 28979 tablet_bootstrap.cc:654] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:35.746735 29002 ts_tablet_manager.cc:1918] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:35.747244 29002 log.cc:1196] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/d544e1893e27454a8b482fde7af44102
I20260925 01:17:35.748045 29002 ts_tablet_manager.cc:1939] T d544e1893e27454a8b482fde7af44102 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Deleting consensus metadata
I20260925 01:17:35.748831 28978 tablet_bootstrap.cc:492] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef: Bootstrap starting.
I20260925 01:17:35.753082 28978 tablet_bootstrap.cc:654] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:35.754575 28454 catalog_manager.cc:4841] TS 39928671a1cf4d6dbdc6a4e7ff1071ef (127.20.102.130:44795): tablet d544e1893e27454a8b482fde7af44102 (table test-table [id=0c0b291bac95445d828c0eb45bcd9a62]) successfully deleted
I20260925 01:17:35.754819 29003 tablet_replica.cc:330] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70: stopping tablet replica
I20260925 01:17:35.755016 28979 tablet_bootstrap.cc:492] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70: No bootstrap required, opened a new log
I20260925 01:17:35.755175 28979 ts_tablet_manager.cc:1397] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.007s
I20260925 01:17:35.755354 29003 raft_consensus.cc:2229] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:35.755831 29003 raft_consensus.cc:2258] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:35.756331 28979 raft_consensus.cc:348] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:35.756584 28979 raft_consensus.cc:374] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:35.756621 29004 tablet_replica.cc:330] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644: stopping tablet replica
I20260925 01:17:35.756834 28979 raft_consensus.cc:729] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b86efccdf27348cb8230e37b637d9e70, State: Initialized, Role: FOLLOWER
I20260925 01:17:35.757061 29004 raft_consensus.cc:2229] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:17:35.757699 29004 raft_consensus.cc:2258] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:35.757867 28979 consensus_queue.cc:260] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:35.758404 28979 ts_tablet_manager.cc:1428] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 01:17:35.758630 28982 tablet_bootstrap.cc:492] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644: Bootstrap starting.
I20260925 01:17:35.759095 28978 tablet_bootstrap.cc:492] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef: No bootstrap required, opened a new log
I20260925 01:17:35.759207 28978 ts_tablet_manager.cc:1397] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20260925 01:17:35.759722 28978 raft_consensus.cc:348] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:35.759862 28978 raft_consensus.cc:374] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:35.759918 28978 raft_consensus.cc:729] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39928671a1cf4d6dbdc6a4e7ff1071ef, State: Initialized, Role: FOLLOWER
I20260925 01:17:35.760202 28982 tablet_bootstrap.cc:654] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:35.760159 28978 consensus_queue.cc:260] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:35.762145 29004 ts_tablet_manager.cc:1905] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:35.762859 28978 ts_tablet_manager.cc:1428] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260925 01:17:35.766892 28982 tablet_bootstrap.cc:492] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644: No bootstrap required, opened a new log
I20260925 01:17:35.767053 28982 ts_tablet_manager.cc:1397] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260925 01:17:35.767946 28982 raft_consensus.cc:348] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:35.768167 28982 raft_consensus.cc:374] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:35.768265 28982 raft_consensus.cc:729] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42f998c7c4947259fd79fdfd9ff7644, State: Initialized, Role: FOLLOWER
I20260925 01:17:35.768570 28982 consensus_queue.cc:260] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:35.769434 29004 ts_tablet_manager.cc:1918] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:35.769593 29004 log.cc:1196] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/d544e1893e27454a8b482fde7af44102
I20260925 01:17:35.770157 29004 ts_tablet_manager.cc:1939] T d544e1893e27454a8b482fde7af44102 P c42f998c7c4947259fd79fdfd9ff7644: Deleting consensus metadata
I20260925 01:17:35.770171 29003 ts_tablet_manager.cc:1905] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:35.772194 28455 catalog_manager.cc:4841] TS c42f998c7c4947259fd79fdfd9ff7644 (127.20.102.129:36191): tablet d544e1893e27454a8b482fde7af44102 (table test-table [id=0c0b291bac95445d828c0eb45bcd9a62]) successfully deleted
I20260925 01:17:35.773339 28982 ts_tablet_manager.cc:1428] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260925 01:17:35.775830 29003 ts_tablet_manager.cc:1918] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:35.776002 29003 log.cc:1196] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/d544e1893e27454a8b482fde7af44102
I20260925 01:17:35.776467 29003 ts_tablet_manager.cc:1939] T d544e1893e27454a8b482fde7af44102 P b86efccdf27348cb8230e37b637d9e70: Deleting consensus metadata
I20260925 01:17:35.778275 28455 catalog_manager.cc:4841] TS b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847): tablet d544e1893e27454a8b482fde7af44102 (table test-table [id=0c0b291bac95445d828c0eb45bcd9a62]) successfully deleted
I20260925 01:17:36.176173 29000 raft_consensus.cc:482] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:17:36.176369 29000 raft_consensus.cc:504] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.177078 29000 leader_election.cc:288] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39928671a1cf4d6dbdc6a4e7ff1071ef (127.20.102.130:44795), b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847)
I20260925 01:17:36.177713 28903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18fb525397f421492791136f849094d" candidate_uuid: "c42f998c7c4947259fd79fdfd9ff7644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b86efccdf27348cb8230e37b637d9e70" is_pre_election: true
I20260925 01:17:36.177747 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18fb525397f421492791136f849094d" candidate_uuid: "c42f998c7c4947259fd79fdfd9ff7644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" is_pre_election: true
I20260925 01:17:36.177971 28903 raft_consensus.cc:2454] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c42f998c7c4947259fd79fdfd9ff7644 in term 0.
I20260925 01:17:36.177983 28765 raft_consensus.cc:2454] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c42f998c7c4947259fd79fdfd9ff7644 in term 0.
I20260925 01:17:36.178454 28559 leader_election.cc:302] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b86efccdf27348cb8230e37b637d9e70, c42f998c7c4947259fd79fdfd9ff7644; no voters:
I20260925 01:17:36.178824 29000 raft_consensus.cc:2781] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:17:36.178961 29000 raft_consensus.cc:482] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:17:36.179075 29000 raft_consensus.cc:3037] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:36.181324 29000 raft_consensus.cc:504] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.181792 29000 leader_election.cc:288] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [CANDIDATE]: Term 1 election: Requested vote from peers 39928671a1cf4d6dbdc6a4e7ff1071ef (127.20.102.130:44795), b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847)
I20260925 01:17:36.182307 28903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18fb525397f421492791136f849094d" candidate_uuid: "c42f998c7c4947259fd79fdfd9ff7644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b86efccdf27348cb8230e37b637d9e70"
I20260925 01:17:36.182325 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b18fb525397f421492791136f849094d" candidate_uuid: "c42f998c7c4947259fd79fdfd9ff7644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef"
I20260925 01:17:36.182511 28903 raft_consensus.cc:3037] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:36.182533 28765 raft_consensus.cc:3037] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:36.184455 28903 raft_consensus.cc:2454] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c42f998c7c4947259fd79fdfd9ff7644 in term 1.
I20260925 01:17:36.184530 28765 raft_consensus.cc:2454] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c42f998c7c4947259fd79fdfd9ff7644 in term 1.
I20260925 01:17:36.184926 28559 leader_election.cc:302] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b86efccdf27348cb8230e37b637d9e70, c42f998c7c4947259fd79fdfd9ff7644; no voters:
I20260925 01:17:36.185322 29000 raft_consensus.cc:2781] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:36.185489 29000 raft_consensus.cc:686] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [term 1 LEADER]: Becoming Leader. State: Replica: c42f998c7c4947259fd79fdfd9ff7644, State: Running, Role: LEADER
I20260925 01:17:36.185750 29000 consensus_queue.cc:237] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.188376 28467 catalog_manager.cc:5495] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 reported cstate change: term changed from 0 to 1, leader changed from <none> to c42f998c7c4947259fd79fdfd9ff7644 (127.20.102.129). New cstate: current_term: 1 leader_uuid: "c42f998c7c4947259fd79fdfd9ff7644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } health_report { overall_health: UNKNOWN } } }
Sep 25 01:17:36 dist-test-slave-s12s krb5kdc[28423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299053, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.190@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
I20260925 01:17:36.255951 29014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.238870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43422 (local address 127.20.102.190:41721)
0925 01:17:36.239785 (+ 915us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:36.239793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:36.239962 (+ 169us) server_negotiation.cc:407] Connection header received
0925 01:17:36.240168 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:36.240173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:36.240237 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:36.240350 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:36.241623 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.242370 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.243629 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.244033 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.244179 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.250650 (+ 6471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:36.250655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:36.250658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:36.250686 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:36.253133 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:36.253809 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:36.253816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:36.253819 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:36.253936 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:36.254297 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:36.254304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:36.254308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:36.254693 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:36.254853 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:36.255266 (+ 413us) server_negotiation.cc:299] Negotiation successful
0925 01:17:36.255523 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":731,"thread_start_us":519,"threads_started":1}
I20260925 01:17:36.256024 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.238894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:41721 (local address 127.0.0.1:43422)
0925 01:17:36.239848 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:36.239877 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:36.240040 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:36.240372 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:36.240376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:36.241250 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:36.241458 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.241468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.242511 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.242517 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.243478 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.243490 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.244196 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.244202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.244360 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.248097 (+ 3737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:36.248115 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:36.250444 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:36.253276 (+ 2832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:36.253288 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:36.253296 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:36.253690 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:36.254052 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:36.254057 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:36.254060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:36.254191 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:36.254821 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:36.254831 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:36.255001 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 01:17:36.255192 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 01:17:36.255633 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":167,"thread_start_us":562,"threads_started":1}
I20260925 01:17:36.261878 28467 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:43422:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 01:17:36.262918 28467 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:17:36.269690 28745 tablet_service.cc:1467] Processing CreateTablet for tablet 4ceaeae9a5944737a71459fbde7bdb24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83238e571aea4386bde2d6ef0dcc58a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:17:36.269856 28605 tablet_service.cc:1467] Processing CreateTablet for tablet 4ceaeae9a5944737a71459fbde7bdb24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83238e571aea4386bde2d6ef0dcc58a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:17:36.269832 28884 tablet_service.cc:1467] Processing CreateTablet for tablet 4ceaeae9a5944737a71459fbde7bdb24 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=83238e571aea4386bde2d6ef0dcc58a5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 01:17:36.270287 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ceaeae9a5944737a71459fbde7bdb24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:36.270352 28605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ceaeae9a5944737a71459fbde7bdb24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:36.270370 28884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ceaeae9a5944737a71459fbde7bdb24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:36.277583 29016 tablet_bootstrap.cc:492] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Bootstrap starting.
I20260925 01:17:36.279407 29016 tablet_bootstrap.cc:654] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:36.281358 29016 tablet_bootstrap.cc:492] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef: No bootstrap required, opened a new log
I20260925 01:17:36.281531 29016 ts_tablet_manager.cc:1397] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 01:17:36.281889 29018 tablet_bootstrap.cc:492] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70: Bootstrap starting.
I20260925 01:17:36.282191 29016 raft_consensus.cc:348] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.282400 29016 raft_consensus.cc:374] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:36.282539 29016 raft_consensus.cc:729] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39928671a1cf4d6dbdc6a4e7ff1071ef, State: Initialized, Role: FOLLOWER
I20260925 01:17:36.282776 29016 consensus_queue.cc:260] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.284188 29018 tablet_bootstrap.cc:654] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:36.285349 29017 tablet_bootstrap.cc:492] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644: Bootstrap starting.
I20260925 01:17:36.286615 29016 ts_tablet_manager.cc:1428] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260925 01:17:36.287202 29017 tablet_bootstrap.cc:654] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:36.287614 29018 tablet_bootstrap.cc:492] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70: No bootstrap required, opened a new log
I20260925 01:17:36.287765 29018 ts_tablet_manager.cc:1397] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 01:17:36.288882 29018 raft_consensus.cc:348] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.289134 29018 raft_consensus.cc:374] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:36.289354 29018 raft_consensus.cc:729] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b86efccdf27348cb8230e37b637d9e70, State: Initialized, Role: FOLLOWER
I20260925 01:17:36.290015 29017 tablet_bootstrap.cc:492] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644: No bootstrap required, opened a new log
I20260925 01:17:36.289824 29018 consensus_queue.cc:260] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.290256 29017 ts_tablet_manager.cc:1397] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 01:17:36.290819 29020 tablet_replica.cc:439] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } }
I20260925 01:17:36.291054 29017 raft_consensus.cc:348] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.291285 29017 raft_consensus.cc:374] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:36.291555 29017 raft_consensus.cc:729] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42f998c7c4947259fd79fdfd9ff7644, State: Initialized, Role: FOLLOWER
I20260925 01:17:36.291890 29017 consensus_queue.cc:260] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.292280 29018 ts_tablet_manager.cc:1428] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260925 01:17:36.292375 29000 tablet_replica.cc:439] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } }
I20260925 01:17:36.292631 29020 tablet_replica.cc:442] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:17:36.292666 29000 tablet_replica.cc:442] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:17:36.292896 29020 raft_consensus.cc:482] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:17:36.293089 29020 raft_consensus.cc:504] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.293944 29017 ts_tablet_manager.cc:1428] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 01:17:36.296800 29024 tablet_replica.cc:439] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } }
I20260925 01:17:36.298084 29024 tablet_replica.cc:442] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:17:36.300633 29020 leader_election.cc:288] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c42f998c7c4947259fd79fdfd9ff7644 (127.20.102.129:36191), b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847)
I20260925 01:17:36.308809 29026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.295693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:59385 (local address 127.20.102.129:36191)
0925 01:17:36.300552 (+ 4859us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:36.300558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:36.300577 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:17:36.300647 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:36.300672 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:36.300796 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:36.301754 (+ 958us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:36.302006 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.303501 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.306501 (+ 3000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.307688 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.307823 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.307874 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:36.308304 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 01:17:36.308406 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4743,"thread_start_us":4621,"threads_started":1}
I20260925 01:17:36.308841 29025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.295339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:36191 (local address 127.20.102.130:59385)
0925 01:17:36.295970 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:36.296002 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:36.296137 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:36.300986 (+ 4849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:36.300994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:36.301035 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:36.301545 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.301559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.303705 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.303714 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.306293 (+ 2579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.306310 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.307860 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.307868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.308134 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.308170 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:17:36.308268 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 01:17:36.308397 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":473,"threads_started":1}
I20260925 01:17:36.309813 28629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceaeae9a5944737a71459fbde7bdb24" candidate_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42f998c7c4947259fd79fdfd9ff7644" is_pre_election: true
I20260925 01:17:36.310259 28629 raft_consensus.cc:2454] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39928671a1cf4d6dbdc6a4e7ff1071ef in term 0.
I20260925 01:17:36.310850 28699 leader_election.cc:302] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39928671a1cf4d6dbdc6a4e7ff1071ef, c42f998c7c4947259fd79fdfd9ff7644; no voters:
I20260925 01:17:36.312312 29023 raft_consensus.cc:2781] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:17:36.312454 29023 raft_consensus.cc:482] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:17:36.312619 29023 raft_consensus.cc:3037] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:36.313145 29027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.300978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:36847 (local address 127.20.102.130:47423)
0925 01:17:36.306282 (+ 5304us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:36.306300 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:36.306437 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:36.306922 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:36.306926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:36.306954 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:36.307280 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.307291 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.308727 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.308733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.310275 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.310290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.312138 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.312145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.312379 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.312414 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:17:36.312519 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 01:17:36.312653 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5168,"thread_start_us":5041,"threads_started":1}
I20260925 01:17:36.313478 29028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.301383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:47423 (local address 127.20.102.131:36847)
0925 01:17:36.306580 (+ 5197us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:36.306584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:36.306605 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:17:36.306655 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:36.306658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:36.306708 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:36.306906 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:36.307555 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.308565 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.310438 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.311898 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.312100 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.312174 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:36.312626 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 01:17:36.312865 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5074,"thread_start_us":4947,"threads_started":1}
I20260925 01:17:36.314014 28903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceaeae9a5944737a71459fbde7bdb24" candidate_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b86efccdf27348cb8230e37b637d9e70" is_pre_election: true
I20260925 01:17:36.314278 28903 raft_consensus.cc:2454] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39928671a1cf4d6dbdc6a4e7ff1071ef in term 0.
I20260925 01:17:36.315532 29023 raft_consensus.cc:504] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.316187 29023 leader_election.cc:288] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [CANDIDATE]: Term 1 election: Requested vote from peers c42f998c7c4947259fd79fdfd9ff7644 (127.20.102.129:36191), b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847)
I20260925 01:17:36.316744 28629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceaeae9a5944737a71459fbde7bdb24" candidate_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42f998c7c4947259fd79fdfd9ff7644"
I20260925 01:17:36.316869 28903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ceaeae9a5944737a71459fbde7bdb24" candidate_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b86efccdf27348cb8230e37b637d9e70"
I20260925 01:17:36.316938 28629 raft_consensus.cc:3037] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:36.317029 28903 raft_consensus.cc:3037] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:36.319063 28903 raft_consensus.cc:2454] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39928671a1cf4d6dbdc6a4e7ff1071ef in term 1.
I20260925 01:17:36.319453 28699 leader_election.cc:302] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39928671a1cf4d6dbdc6a4e7ff1071ef, b86efccdf27348cb8230e37b637d9e70; no voters:
I20260925 01:17:36.319725 29023 raft_consensus.cc:2781] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:36.319900 29023 raft_consensus.cc:686] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 LEADER]: Becoming Leader. State: Replica: 39928671a1cf4d6dbdc6a4e7ff1071ef, State: Running, Role: LEADER
I20260925 01:17:36.320040 28629 raft_consensus.cc:2454] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39928671a1cf4d6dbdc6a4e7ff1071ef in term 1.
I20260925 01:17:36.320302 29023 consensus_queue.cc:237] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.328913 29020 tablet_replica.cc:439] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef: TxnStatusTablet state changed. Reason: New leader 39928671a1cf4d6dbdc6a4e7ff1071ef. Latest consensus state: current_term: 1 leader_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } }
I20260925 01:17:36.329241 29020 tablet_replica.cc:442] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:17:36.329921 29030 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 01:17:36.330205 29030 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 01:17:36.330464 28467 catalog_manager.cc:5495] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 39928671a1cf4d6dbdc6a4e7ff1071ef (127.20.102.130). New cstate: current_term: 1 leader_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } health_report { overall_health: UNKNOWN } } }
I20260925 01:17:36.350896 29015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.341639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:44795 (local address 127.0.0.1:46268)
0925 01:17:36.341836 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:36.341859 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:36.342028 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:36.345566 (+ 3538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:36.345572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:36.345590 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:17:36.345807 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.345818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.347086 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.347093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.348279 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.348291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.349106 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.349111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.349292 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.349347 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:17:36.350319 (+ 972us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:17:36.350341 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 01:17:36.350421 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 01:17:36.350584 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 01:17:36.350926 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.341723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46268 (local address 127.20.102.130:44795)
0925 01:17:36.345185 (+ 3462us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:36.345189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:36.345208 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:17:36.345281 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:36.345284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:36.345356 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:36.345531 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:17:36.345981 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.346943 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.348428 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.348944 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.349093 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.349587 (+ 494us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:17:36.350157 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:17:36.350381 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:36.350451 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 01:17:36.350532 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3353,"thread_start_us":3187,"threads_started":1}
I20260925 01:17:36.355680 28903 raft_consensus.cc:1261] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Refusing update from remote peer 39928671a1cf4d6dbdc6a4e7ff1071ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:36.355742 28629 raft_consensus.cc:1261] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Refusing update from remote peer 39928671a1cf4d6dbdc6a4e7ff1071ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:36.356438 29023 consensus_queue.cc:1035] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:36.356839 29020 consensus_queue.cc:1035] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:36.362406 29000 tablet_replica.cc:439] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644: TxnStatusTablet state changed. Reason: New leader 39928671a1cf4d6dbdc6a4e7ff1071ef. Latest consensus state: current_term: 1 leader_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } }
I20260925 01:17:36.362705 29000 tablet_replica.cc:442] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:17:36.380759 29023 tablet_replica.cc:439] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } }
I20260925 01:17:36.381122 29023 tablet_replica.cc:442] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:17:36.384898 29000 tablet_replica.cc:439] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } }
I20260925 01:17:36.385200 29000 tablet_replica.cc:442] T 4ceaeae9a5944737a71459fbde7bdb24 P c42f998c7c4947259fd79fdfd9ff7644: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:17:36.388347 29024 tablet_replica.cc:439] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70: TxnStatusTablet state changed. Reason: New leader 39928671a1cf4d6dbdc6a4e7ff1071ef. Latest consensus state: current_term: 1 leader_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } }
I20260925 01:17:36.388701 29024 tablet_replica.cc:442] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:17:36.393659 29041 tablet_replica.cc:439] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } }
I20260925 01:17:36.394148 29041 tablet_replica.cc:442] T 4ceaeae9a5944737a71459fbde7bdb24 P 39928671a1cf4d6dbdc6a4e7ff1071ef: This TxnStatusTablet replica's current role is: LEADER
I20260925 01:17:36.397651 29024 tablet_replica.cc:439] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } } }
I20260925 01:17:36.397905 29024 tablet_replica.cc:442] T 4ceaeae9a5944737a71459fbde7bdb24 P b86efccdf27348cb8230e37b637d9e70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 01:17:36.413362 29045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.406549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:44795 (local address 127.20.102.129:57807)
0925 01:17:36.407141 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:36.407160 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:36.407341 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:36.407734 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:36.407741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:36.407759 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:17:36.408053 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.408063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.409526 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.409532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.411166 (+ 1634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.411180 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.412199 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.412204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.412327 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.412381 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:17:36.412843 (+ 462us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:17:36.412853 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 01:17:36.412920 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 01:17:36.413081 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":239,"threads_started":1}
I20260925 01:17:36.413383 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.406411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:57807 (local address 127.20.102.130:44795)
0925 01:17:36.406625 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:36.406632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:36.407255 (+ 623us) server_negotiation.cc:407] Connection header received
0925 01:17:36.407482 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:36.407488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:36.407585 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:36.407685 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:17:36.408199 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.409365 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.411331 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.412040 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.412166 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.412565 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:17:36.412754 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:17:36.412844 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:36.412975 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 01:17:36.413070 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 01:17:36.424042 28765 raft_consensus.cc:1261] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Refusing update from remote peer c42f998c7c4947259fd79fdfd9ff7644: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:36.424032 28903 raft_consensus.cc:1261] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Refusing update from remote peer c42f998c7c4947259fd79fdfd9ff7644: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:36.424712 29000 consensus_queue.cc:1035] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:36.425001 28989 consensus_queue.cc:1035] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:36.476092 29026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.465626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:48077 (local address 127.20.102.129:36191)
0925 01:17:36.465886 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:36.465896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:36.469508 (+ 3612us) server_negotiation.cc:407] Connection header received
0925 01:17:36.469866 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:36.469872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:36.469965 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:36.470072 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:17:36.470847 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.471962 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.473918 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.474654 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.474781 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.475344 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:17:36.475476 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:17:36.475583 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:36.475724 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 01:17:36.475809 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 01:17:36.476162 29052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.464936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:36191 (local address 127.20.102.130:48077)
0925 01:17:36.469382 (+ 4446us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:36.469409 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:36.469699 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:36.470264 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:36.470271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:36.470284 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:17:36.470666 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.470676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.472113 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.472117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.473759 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.473780 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.474857 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.474864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.475075 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.475157 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:17:36.475596 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:17:36.475606 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 01:17:36.475677 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 01:17:36.475787 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4188,"thread_start_us":4020,"threads_started":1}
I20260925 01:17:36.519805 28468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:36.520079 28468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:36.523798 28468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9: Sending DeleteTablet for 3 replicas of tablet b18fb525397f421492791136f849094d
I20260925 01:17:36.524725 28605 tablet_service.cc:1514] Processing DeleteTablet for tablet b18fb525397f421492791136f849094d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:36 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:51324
I20260925 01:17:36.524799 28745 tablet_service.cc:1514] Processing DeleteTablet for tablet b18fb525397f421492791136f849094d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:36 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:46266
I20260925 01:17:36.524987 28884 tablet_service.cc:1514] Processing DeleteTablet for tablet b18fb525397f421492791136f849094d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:36 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:41724
I20260925 01:17:36.535364 29056 tablet_replica.cc:330] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70: stopping tablet replica
I20260925 01:17:36.536293 29056 raft_consensus.cc:2229] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:36.537020 29056 raft_consensus.cc:2258] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:36.538811 29055 tablet_replica.cc:330] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef: stopping tablet replica
I20260925 01:17:36.539029 29055 raft_consensus.cc:2229] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:36.539333 29055 raft_consensus.cc:2258] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:36.538851 29056 ts_tablet_manager.cc:1905] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:36.540406 29055 ts_tablet_manager.cc:1905] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:36.541937 29054 tablet_replica.cc:330] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644: stopping tablet replica
I20260925 01:17:36.542140 29054 raft_consensus.cc:2229] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:17:36.542590 29054 raft_consensus.cc:2258] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:36.543649 29054 ts_tablet_manager.cc:1905] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:36.547857 29056 ts_tablet_manager.cc:1918] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:17:36.548079 29056 log.cc:1196] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/b18fb525397f421492791136f849094d
I20260925 01:17:36.548156 29055 ts_tablet_manager.cc:1918] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:17:36.548307 29055 log.cc:1196] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/b18fb525397f421492791136f849094d
I20260925 01:17:36.548766 29056 ts_tablet_manager.cc:1939] T b18fb525397f421492791136f849094d P b86efccdf27348cb8230e37b637d9e70: Deleting consensus metadata
I20260925 01:17:36.548857 29055 ts_tablet_manager.cc:1939] T b18fb525397f421492791136f849094d P 39928671a1cf4d6dbdc6a4e7ff1071ef: Deleting consensus metadata
I20260925 01:17:36.550079 28454 catalog_manager.cc:4841] TS 39928671a1cf4d6dbdc6a4e7ff1071ef (127.20.102.130:44795): tablet b18fb525397f421492791136f849094d (table test-table [id=e0e5d2f9236142d5bdcfb0ae9d5bf41a]) successfully deleted
I20260925 01:17:36.550081 28455 catalog_manager.cc:4841] TS b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847): tablet b18fb525397f421492791136f849094d (table test-table [id=e0e5d2f9236142d5bdcfb0ae9d5bf41a]) successfully deleted
I20260925 01:17:36.550967 29054 ts_tablet_manager.cc:1918] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:17:36.551123 29054 log.cc:1196] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/b18fb525397f421492791136f849094d
I20260925 01:17:36.551539 29054 ts_tablet_manager.cc:1939] T b18fb525397f421492791136f849094d P c42f998c7c4947259fd79fdfd9ff7644: Deleting consensus metadata
I20260925 01:17:36.552527 28455 catalog_manager.cc:4841] TS c42f998c7c4947259fd79fdfd9ff7644 (127.20.102.129:36191): tablet b18fb525397f421492791136f849094d (table test-table [id=e0e5d2f9236142d5bdcfb0ae9d5bf41a]) successfully deleted
Sep 25 01:17:36 dist-test-slave-s12s krb5kdc[28423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 01:17:36 dist-test-slave-s12s krb5kdc[28423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:36 dist-test-slave-s12s krb5kdc[28423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:17:36.604009 29014 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:36.591208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.20.102.190:41721)
0925 01:17:36.591501 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:36.591508 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:36.591904 (+ 396us) server_negotiation.cc:407] Connection header received
0925 01:17:36.592123 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:36.592127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:36.592211 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:36.592314 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:36.594353 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.595190 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.597222 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.597679 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.597778 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.603701 (+ 5923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43432: BlockingRecv error: recv got EOF from 127.0.0.1:43432 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 01:17:36 dist-test-slave-s12s krb5kdc[28423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM
I20260925 01:17:36.621869 29014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.606947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43448 (local address 127.20.102.190:41721)
0925 01:17:36.607165 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:36.607175 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:36.607540 (+ 365us) server_negotiation.cc:407] Connection header received
0925 01:17:36.607774 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:36.607779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:36.607855 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:36.607962 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:36.609242 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.609995 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.611234 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.611672 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.611814 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.616299 (+ 4485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:36.616306 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:36.616316 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:36.616356 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:36.619121 (+ 2765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:36.619791 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:36.619798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:36.619801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:36.619943 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:36.620358 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:36.620364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:36.620367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:36.620656 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:36.620983 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:36.621278 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 01:17:36.621512 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:17:36.625801 28468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:17:36.626418 28468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:17:36.633334 28605 tablet_service.cc:1467] Processing CreateTablet for tablet f1ade0a46d3e4227bb8720df072e1296 (DEFAULT_TABLE table=test-table [id=2b5f7c18463840d7afd000891ce5a578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:36.633845 28605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ade0a46d3e4227bb8720df072e1296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:36.633796 28745 tablet_service.cc:1467] Processing CreateTablet for tablet f1ade0a46d3e4227bb8720df072e1296 (DEFAULT_TABLE table=test-table [id=2b5f7c18463840d7afd000891ce5a578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:36.634264 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ade0a46d3e4227bb8720df072e1296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:36.637136 28884 tablet_service.cc:1467] Processing CreateTablet for tablet f1ade0a46d3e4227bb8720df072e1296 (DEFAULT_TABLE table=test-table [id=2b5f7c18463840d7afd000891ce5a578]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:36.637547 28884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ade0a46d3e4227bb8720df072e1296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:36.639437 29017 tablet_bootstrap.cc:492] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644: Bootstrap starting.
I20260925 01:17:36.641258 29017 tablet_bootstrap.cc:654] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:36.641561 29016 tablet_bootstrap.cc:492] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Bootstrap starting.
I20260925 01:17:36.643201 29016 tablet_bootstrap.cc:654] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:36.643523 29018 tablet_bootstrap.cc:492] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70: Bootstrap starting.
I20260925 01:17:36.643539 29017 tablet_bootstrap.cc:492] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644: No bootstrap required, opened a new log
I20260925 01:17:36.643734 29017 ts_tablet_manager.cc:1397] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 01:17:36.644688 29017 raft_consensus.cc:348] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.645067 29017 raft_consensus.cc:374] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:36.645098 29018 tablet_bootstrap.cc:654] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:36.645165 29017 raft_consensus.cc:729] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42f998c7c4947259fd79fdfd9ff7644, State: Initialized, Role: FOLLOWER
I20260925 01:17:36.645531 29016 tablet_bootstrap.cc:492] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef: No bootstrap required, opened a new log
I20260925 01:17:36.645473 29017 consensus_queue.cc:260] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.645706 29016 ts_tablet_manager.cc:1397] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 01:17:36.646018 29017 ts_tablet_manager.cc:1428] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 01:17:36.646733 29016 raft_consensus.cc:348] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.647123 29016 raft_consensus.cc:374] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:36.647306 29016 raft_consensus.cc:729] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39928671a1cf4d6dbdc6a4e7ff1071ef, State: Initialized, Role: FOLLOWER
I20260925 01:17:36.647670 29016 consensus_queue.cc:260] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.648237 29016 ts_tablet_manager.cc:1428] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 01:17:36.657823 29018 tablet_bootstrap.cc:492] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70: No bootstrap required, opened a new log
I20260925 01:17:36.658094 29018 ts_tablet_manager.cc:1397] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260925 01:17:36.658825 29018 raft_consensus.cc:348] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.659065 29018 raft_consensus.cc:374] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:36.659161 29018 raft_consensus.cc:729] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b86efccdf27348cb8230e37b637d9e70, State: Initialized, Role: FOLLOWER
I20260925 01:17:36.659427 29018 consensus_queue.cc:260] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.660843 29018 ts_tablet_manager.cc:1428] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 01:17:36.671852 29024 raft_consensus.cc:482] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:17:36.672091 29024 raft_consensus.cc:504] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.674249 29024 leader_election.cc:288] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c42f998c7c4947259fd79fdfd9ff7644 (127.20.102.129:36191), 39928671a1cf4d6dbdc6a4e7ff1071ef (127.20.102.130:44795)
I20260925 01:17:36.679821 28989 raft_consensus.cc:482] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:17:36.680050 28989 raft_consensus.cc:504] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.680867 28989 leader_election.cc:288] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39928671a1cf4d6dbdc6a4e7ff1071ef (127.20.102.130:44795), b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847)
I20260925 01:17:36.682116 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ade0a46d3e4227bb8720df072e1296" candidate_uuid: "c42f998c7c4947259fd79fdfd9ff7644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" is_pre_election: true
I20260925 01:17:36.682407 28765 raft_consensus.cc:2454] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c42f998c7c4947259fd79fdfd9ff7644 in term 0.
I20260925 01:17:36.682973 28904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ade0a46d3e4227bb8720df072e1296" candidate_uuid: "c42f998c7c4947259fd79fdfd9ff7644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b86efccdf27348cb8230e37b637d9e70" is_pre_election: true
I20260925 01:17:36.683248 28904 raft_consensus.cc:2454] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c42f998c7c4947259fd79fdfd9ff7644 in term 0.
I20260925 01:17:36.683840 28559 leader_election.cc:302] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b86efccdf27348cb8230e37b637d9e70, c42f998c7c4947259fd79fdfd9ff7644; no voters:
I20260925 01:17:36.684571 28989 raft_consensus.cc:2781] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:17:36.684724 28989 raft_consensus.cc:482] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:17:36.684834 28989 raft_consensus.cc:3037] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:36.688594 28989 raft_consensus.cc:504] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.689929 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ade0a46d3e4227bb8720df072e1296" candidate_uuid: "c42f998c7c4947259fd79fdfd9ff7644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef"
I20260925 01:17:36.690234 28765 raft_consensus.cc:3037] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:36.693670 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.675299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:54929 (local address 127.20.102.130:44795)
0925 01:17:36.675739 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:36.675748 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:36.685537 (+ 9789us) server_negotiation.cc:407] Connection header received
0925 01:17:36.685969 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:36.685975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:36.686067 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:36.686145 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:36.687000 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.688151 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.690920 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.692396 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.692539 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.692631 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:36.693218 (+ 587us) server_negotiation.cc:299] Negotiation successful
0925 01:17:36.693367 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281}
I20260925 01:17:36.693789 28765 raft_consensus.cc:2454] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c42f998c7c4947259fd79fdfd9ff7644 in term 1.
I20260925 01:17:36.694273 28558 leader_election.cc:302] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39928671a1cf4d6dbdc6a4e7ff1071ef, c42f998c7c4947259fd79fdfd9ff7644; no voters:
I20260925 01:17:36.694625 29049 raft_consensus.cc:2781] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:36.694840 29049 raft_consensus.cc:686] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 LEADER]: Becoming Leader. State: Replica: c42f998c7c4947259fd79fdfd9ff7644, State: Running, Role: LEADER
I20260925 01:17:36.695266 29026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.675047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:57697 (local address 127.20.102.129:36191)
0925 01:17:36.675551 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:36.675562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:36.685348 (+ 9786us) server_negotiation.cc:407] Connection header received
0925 01:17:36.685662 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:36.685667 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:36.685779 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:36.685848 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:17:36.686774 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.688017 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.692784 (+ 4767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.693959 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.694097 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.694164 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:36.694793 (+ 629us) server_negotiation.cc:299] Negotiation successful
0925 01:17:36.694952 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336}
I20260925 01:17:36.695151 29049 consensus_queue.cc:237] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.695873 28904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ade0a46d3e4227bb8720df072e1296" candidate_uuid: "c42f998c7c4947259fd79fdfd9ff7644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b86efccdf27348cb8230e37b637d9e70"
I20260925 01:17:36.696112 28904 raft_consensus.cc:3037] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:36.699232 28904 raft_consensus.cc:2454] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c42f998c7c4947259fd79fdfd9ff7644 in term 1.
I20260925 01:17:36.699903 28989 leader_election.cc:288] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [CANDIDATE]: Term 1 election: Requested vote from peers 39928671a1cf4d6dbdc6a4e7ff1071ef (127.20.102.130:44795), b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847)
I20260925 01:17:36.700515 29073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.674568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:36191 (local address 127.20.102.131:57697)
0925 01:17:36.685217 (+ 10649us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:36.685244 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:36.685482 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:36.686083 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:36.686088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:36.686117 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:36.686552 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.686570 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.688224 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.688229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.690600 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.690615 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.694426 (+ 3811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.694431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.694591 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.694632 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 01:17:36.695538 (+ 906us) client_negotiation.cc:240] Negotiation successful
0925 01:17:36.695668 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10473,"thread_start_us":10985,"threads_started":1}
I20260925 01:17:36.701265 28629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ade0a46d3e4227bb8720df072e1296" candidate_uuid: "b86efccdf27348cb8230e37b637d9e70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42f998c7c4947259fd79fdfd9ff7644" is_pre_election: true
I20260925 01:17:36.702509 29072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.674751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:44795 (local address 127.20.102.131:54929)
0925 01:17:36.685438 (+ 10687us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:36.685454 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:36.685779 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:36.686372 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:36.686379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:36.686401 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:17:36.686810 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.686822 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.688353 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.688358 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.690778 (+ 2420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.690790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.692827 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.692835 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.693022 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.693057 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:17:36.694031 (+ 974us) client_negotiation.cc:240] Negotiation successful
0925 01:17:36.694200 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10573,"thread_start_us":13426,"threads_started":1}
I20260925 01:17:36.704217 28765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ade0a46d3e4227bb8720df072e1296" candidate_uuid: "b86efccdf27348cb8230e37b637d9e70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" is_pre_election: true
I20260925 01:17:36.704643 28765 raft_consensus.cc:2379] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b86efccdf27348cb8230e37b637d9e70 in current term 1: Already voted for candidate c42f998c7c4947259fd79fdfd9ff7644 in this term.
I20260925 01:17:36.704442 28467 catalog_manager.cc:5495] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 reported cstate change: term changed from 0 to 1, leader changed from <none> to c42f998c7c4947259fd79fdfd9ff7644 (127.20.102.129). New cstate: current_term: 1 leader_uuid: "c42f998c7c4947259fd79fdfd9ff7644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } health_report { overall_health: UNKNOWN } } }
I20260925 01:17:36.705327 28837 leader_election.cc:302] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b86efccdf27348cb8230e37b637d9e70; no voters: 39928671a1cf4d6dbdc6a4e7ff1071ef, c42f998c7c4947259fd79fdfd9ff7644
I20260925 01:17:36.705893 29024 raft_consensus.cc:2726] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 01:17:36.721184 29026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.716312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51340 (local address 127.20.102.129:36191)
0925 01:17:36.716623 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:36.716629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:36.716649 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:17:36.716797 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:36.716803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:36.716879 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:36.716982 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:17:36.717368 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.718083 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.719399 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.719863 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.720007 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.720432 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:17:36.720551 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:17:36.720654 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:36.720835 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 01:17:36.720913 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260925 01:17:36.724429 28904 raft_consensus.cc:1261] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Refusing update from remote peer c42f998c7c4947259fd79fdfd9ff7644: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:36.724583 28765 raft_consensus.cc:1261] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Refusing update from remote peer c42f998c7c4947259fd79fdfd9ff7644: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:36.725263 29049 consensus_queue.cc:1035] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:36.725795 28989 consensus_queue.cc:1035] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:36.735108 29036 mvcc.cc:205] Tried to move back new op lower bound from 7333064936336613376 to 7333064936226500608. Current Snapshot: MvccSnapshot[applied={T|T < 7333064936336613376 or (T in {7333064936336613376})}]
I20260925 01:17:36.741397 28468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:36.741648 28468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:36.745834 28468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9: Sending DeleteTablet for 3 replicas of tablet f1ade0a46d3e4227bb8720df072e1296
I20260925 01:17:36.746939 28605 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ade0a46d3e4227bb8720df072e1296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:36 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:51324
I20260925 01:17:36.747334 28884 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ade0a46d3e4227bb8720df072e1296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:36 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:41724
I20260925 01:17:36.747651 29054 tablet_replica.cc:330] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644: stopping tablet replica
I20260925 01:17:36.747879 29054 raft_consensus.cc:2229] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:17:36.748224 29054 raft_consensus.cc:2258] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:36.747961 28466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 01:17:36.748509 28883 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ade0a46d3e4227bb8720df072e1296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:36 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:41724
I20260925 01:17:36.748747 29056 tablet_replica.cc:330] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70: stopping tablet replica
W20260925 01:17:36.748749 28466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:17:36.748934 29056 raft_consensus.cc:2229] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:36.749154 29056 raft_consensus.cc:2258] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:36.750479 29054 ts_tablet_manager.cc:1905] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:36.750574 28745 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ade0a46d3e4227bb8720df072e1296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:36 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:46266
I20260925 01:17:36.750957 29055 tablet_replica.cc:330] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef: stopping tablet replica
I20260925 01:17:36.751196 29055 raft_consensus.cc:2229] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:36.751488 29055 raft_consensus.cc:2258] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:36.753577 29055 ts_tablet_manager.cc:1905] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:36.753887 29056 ts_tablet_manager.cc:1905] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:36.761500 29055 ts_tablet_manager.cc:1918] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:36.761646 28883 tablet_service.cc:1467] Processing CreateTablet for tablet 7d6c23453b3b4165ab5352323ceed229 (DEFAULT_TABLE table=test-table [id=b1810258978d414e844162f9a4f954ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:36.761926 29055 log.cc:1196] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/f1ade0a46d3e4227bb8720df072e1296
I20260925 01:17:36.762310 29054 ts_tablet_manager.cc:1918] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:36.762490 29054 log.cc:1196] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/f1ade0a46d3e4227bb8720df072e1296
I20260925 01:17:36.762414 28605 tablet_service.cc:1467] Processing CreateTablet for tablet 7d6c23453b3b4165ab5352323ceed229 (DEFAULT_TABLE table=test-table [id=b1810258978d414e844162f9a4f954ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:36.762954 28605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d6c23453b3b4165ab5352323ceed229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:36.763000 29054 ts_tablet_manager.cc:1939] T f1ade0a46d3e4227bb8720df072e1296 P c42f998c7c4947259fd79fdfd9ff7644: Deleting consensus metadata
I20260925 01:17:36.764190 29055 ts_tablet_manager.cc:1939] T f1ade0a46d3e4227bb8720df072e1296 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Deleting consensus metadata
I20260925 01:17:36.761591 28745 tablet_service.cc:1467] Processing CreateTablet for tablet 7d6c23453b3b4165ab5352323ceed229 (DEFAULT_TABLE table=test-table [id=b1810258978d414e844162f9a4f954ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:17:36.765121 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d6c23453b3b4165ab5352323ceed229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:36.765491 28455 catalog_manager.cc:4841] TS c42f998c7c4947259fd79fdfd9ff7644 (127.20.102.129:36191): tablet f1ade0a46d3e4227bb8720df072e1296 (table test-table [id=2b5f7c18463840d7afd000891ce5a578]) successfully deleted
I20260925 01:17:36.765959 28883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d6c23453b3b4165ab5352323ceed229. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:36.769696 29017 tablet_bootstrap.cc:492] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644: Bootstrap starting.
I20260925 01:17:36.770967 29017 tablet_bootstrap.cc:654] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:36.774336 29016 tablet_bootstrap.cc:492] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Bootstrap starting.
I20260925 01:17:36.774394 28454 catalog_manager.cc:4841] TS 39928671a1cf4d6dbdc6a4e7ff1071ef (127.20.102.130:44795): tablet f1ade0a46d3e4227bb8720df072e1296 (table test-table [id=2b5f7c18463840d7afd000891ce5a578]) successfully deleted
I20260925 01:17:36.775342 29018 tablet_bootstrap.cc:492] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70: Bootstrap starting.
I20260925 01:17:36.775887 29056 ts_tablet_manager.cc:1918] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:17:36.776144 29056 log.cc:1196] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/f1ade0a46d3e4227bb8720df072e1296
I20260925 01:17:36.776190 29016 tablet_bootstrap.cc:654] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:36.776784 29056 ts_tablet_manager.cc:1939] T f1ade0a46d3e4227bb8720df072e1296 P b86efccdf27348cb8230e37b637d9e70: Deleting consensus metadata
I20260925 01:17:36.777350 29018 tablet_bootstrap.cc:654] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:36.778699 28455 catalog_manager.cc:4841] TS b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847): tablet f1ade0a46d3e4227bb8720df072e1296 (table test-table [id=2b5f7c18463840d7afd000891ce5a578]) successfully deleted
I20260925 01:17:36.780048 29017 tablet_bootstrap.cc:492] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644: No bootstrap required, opened a new log
I20260925 01:17:36.780261 29017 ts_tablet_manager.cc:1397] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20260925 01:17:36.781333 29017 raft_consensus.cc:348] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.781598 29017 raft_consensus.cc:374] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:36.781708 29017 raft_consensus.cc:729] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c42f998c7c4947259fd79fdfd9ff7644, State: Initialized, Role: FOLLOWER
I20260925 01:17:36.782560 29017 consensus_queue.cc:260] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.783351 29017 ts_tablet_manager.cc:1428] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260925 01:17:36.784628 28455 catalog_manager.cc:4808] TS b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847): delete failed for tablet f1ade0a46d3e4227bb8720df072e1296 because the tablet was not found. No further retry: Not found: Tablet not found: f1ade0a46d3e4227bb8720df072e1296
I20260925 01:17:36.791181 29018 tablet_bootstrap.cc:492] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70: No bootstrap required, opened a new log
I20260925 01:17:36.791378 29018 ts_tablet_manager.cc:1397] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.007s
I20260925 01:17:36.792146 29018 raft_consensus.cc:348] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.792400 29018 raft_consensus.cc:374] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:36.792541 29018 raft_consensus.cc:729] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b86efccdf27348cb8230e37b637d9e70, State: Initialized, Role: FOLLOWER
I20260925 01:17:36.792870 29018 consensus_queue.cc:260] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.793573 29018 ts_tablet_manager.cc:1428] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 01:17:36.795094 29016 tablet_bootstrap.cc:492] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef: No bootstrap required, opened a new log
I20260925 01:17:36.795310 29016 ts_tablet_manager.cc:1397] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.007s
I20260925 01:17:36.796319 29016 raft_consensus.cc:348] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.796669 29016 raft_consensus.cc:374] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:36.796835 29016 raft_consensus.cc:729] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39928671a1cf4d6dbdc6a4e7ff1071ef, State: Initialized, Role: FOLLOWER
I20260925 01:17:36.797138 29016 consensus_queue.cc:260] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.798002 29016 ts_tablet_manager.cc:1428] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:36.868247 29020 raft_consensus.cc:482] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:17:36.868505 29020 raft_consensus.cc:504] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.869239 29020 leader_election.cc:288] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c42f998c7c4947259fd79fdfd9ff7644 (127.20.102.129:36191), b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847)
I20260925 01:17:36.869881 28904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d6c23453b3b4165ab5352323ceed229" candidate_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b86efccdf27348cb8230e37b637d9e70" is_pre_election: true
I20260925 01:17:36.869941 28629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d6c23453b3b4165ab5352323ceed229" candidate_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42f998c7c4947259fd79fdfd9ff7644" is_pre_election: true
I20260925 01:17:36.870122 28904 raft_consensus.cc:2454] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39928671a1cf4d6dbdc6a4e7ff1071ef in term 0.
I20260925 01:17:36.870272 28629 raft_consensus.cc:2454] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 39928671a1cf4d6dbdc6a4e7ff1071ef in term 0.
I20260925 01:17:36.870612 28699 leader_election.cc:302] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39928671a1cf4d6dbdc6a4e7ff1071ef, b86efccdf27348cb8230e37b637d9e70; no voters:
I20260925 01:17:36.870882 29020 raft_consensus.cc:2781] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:17:36.871042 29020 raft_consensus.cc:482] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:17:36.871183 29020 raft_consensus.cc:3037] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:36.873588 29020 raft_consensus.cc:504] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.874151 29020 leader_election.cc:288] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [CANDIDATE]: Term 1 election: Requested vote from peers c42f998c7c4947259fd79fdfd9ff7644 (127.20.102.129:36191), b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847)
I20260925 01:17:36.874684 28904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d6c23453b3b4165ab5352323ceed229" candidate_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b86efccdf27348cb8230e37b637d9e70"
I20260925 01:17:36.874702 28629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d6c23453b3b4165ab5352323ceed229" candidate_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c42f998c7c4947259fd79fdfd9ff7644"
I20260925 01:17:36.874898 28904 raft_consensus.cc:3037] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:36.874920 28629 raft_consensus.cc:3037] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:36.877853 28629 raft_consensus.cc:2454] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39928671a1cf4d6dbdc6a4e7ff1071ef in term 1.
I20260925 01:17:36.877866 28904 raft_consensus.cc:2454] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39928671a1cf4d6dbdc6a4e7ff1071ef in term 1.
I20260925 01:17:36.878249 28699 leader_election.cc:302] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39928671a1cf4d6dbdc6a4e7ff1071ef, b86efccdf27348cb8230e37b637d9e70; no voters:
I20260925 01:17:36.878531 29020 raft_consensus.cc:2781] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:36.878710 29020 raft_consensus.cc:686] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 LEADER]: Becoming Leader. State: Replica: 39928671a1cf4d6dbdc6a4e7ff1071ef, State: Running, Role: LEADER
I20260925 01:17:36.878999 29020 consensus_queue.cc:237] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } }
I20260925 01:17:36.881834 28466 catalog_manager.cc:5495] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 39928671a1cf4d6dbdc6a4e7ff1071ef (127.20.102.130). New cstate: current_term: 1 leader_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39928671a1cf4d6dbdc6a4e7ff1071ef" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 } health_report { overall_health: UNKNOWN } } }
I20260925 01:17:36.907059 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.901177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46282 (local address 127.20.102.130:44795)
0925 01:17:36.901311 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:36.901315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:36.901462 (+ 147us) server_negotiation.cc:407] Connection header received
0925 01:17:36.901651 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:36.901655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:36.901717 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:36.901788 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:17:36.902232 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.903042 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.905062 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.905606 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.905708 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.906299 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:17:36.906453 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:17:36.906528 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:36.906680 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 01:17:36.906743 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 01:17:36.919025 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.914803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:42421 (local address 127.20.102.130:44795)
0925 01:17:36.914971 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:36.914981 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:36.914998 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:17:36.915203 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:36.915207 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:36.915275 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:36.915347 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:17:36.915680 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.916403 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.917642 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.918070 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.918165 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.918472 (+ 307us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:17:36.918564 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:17:36.918628 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:36.918724 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 01:17:36.918786 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:17:36.919028 29052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.914734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:44795 (local address 127.20.102.130:42421)
0925 01:17:36.914942 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:36.914959 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:36.915076 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:36.915375 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:36.915398 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:36.915413 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 01:17:36.915579 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.915587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.916524 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.916527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.917522 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:36.917534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:36.918194 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:36.918198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:36.918317 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:36.918349 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 01:17:36.918646 (+ 297us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 01:17:36.918654 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 01:17:36.918712 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 01:17:36.918805 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 01:17:36.926847 28904 raft_consensus.cc:1261] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Refusing update from remote peer 39928671a1cf4d6dbdc6a4e7ff1071ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:36.926888 28629 raft_consensus.cc:1261] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Refusing update from remote peer 39928671a1cf4d6dbdc6a4e7ff1071ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:17:36.927387 29023 consensus_queue.cc:1035] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "c42f998c7c4947259fd79fdfd9ff7644" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:36.927734 29020 consensus_queue.cc:1035] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b86efccdf27348cb8230e37b637d9e70" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:17:36.981510 28468 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:36.981786 28468 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:17:36.984964 28468 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d3030c7f7ef40efa3636e077d7a77e9: Sending DeleteTablet for 3 replicas of tablet 7d6c23453b3b4165ab5352323ceed229
I20260925 01:17:36.985901 28745 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d6c23453b3b4165ab5352323ceed229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:36 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:46266
I20260925 01:17:36.985941 28883 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d6c23453b3b4165ab5352323ceed229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:36 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:41724
I20260925 01:17:36.985935 28605 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d6c23453b3b4165ab5352323ceed229 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:17:36 UTC) from {username='oryx', principal='oryx/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:51324
I20260925 01:17:36.986231 29056 tablet_replica.cc:330] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70: stopping tablet replica
I20260925 01:17:36.986263 29055 tablet_replica.cc:330] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef: stopping tablet replica
I20260925 01:17:36.986441 29056 raft_consensus.cc:2229] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:36.986452 29055 raft_consensus.cc:2229] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:17:36.986673 29056 raft_consensus.cc:2258] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:36.987000 29055 raft_consensus.cc:2258] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:36.987319 29054 tablet_replica.cc:330] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644: stopping tablet replica
I20260925 01:17:36.987505 29054 raft_consensus.cc:2229] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:17:36.987717 29054 raft_consensus.cc:2258] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:17:36.987825 29056 ts_tablet_manager.cc:1905] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:36.989356 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 28540
I20260925 01:17:36.989356 29054 ts_tablet_manager.cc:1905] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:36.994992 29055 ts_tablet_manager.cc:1905] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:17:36.995164 29056 ts_tablet_manager.cc:1918] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:17:36.995332 29056 log.cc:1196] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/7d6c23453b3b4165ab5352323ceed229
I20260925 01:17:36.995739 29056 ts_tablet_manager.cc:1939] T 7d6c23453b3b4165ab5352323ceed229 P b86efccdf27348cb8230e37b637d9e70: Deleting consensus metadata
I20260925 01:17:36.996534 29054 ts_tablet_manager.cc:1918] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:17:36.996749 29054 log.cc:1196] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/7d6c23453b3b4165ab5352323ceed229
I20260925 01:17:36.996876 28455 catalog_manager.cc:4841] TS b86efccdf27348cb8230e37b637d9e70 (127.20.102.131:36847): tablet 7d6c23453b3b4165ab5352323ceed229 (table test-table [id=b1810258978d414e844162f9a4f954ea]) successfully deleted
I20260925 01:17:36.997320 29054 ts_tablet_manager.cc:1939] T 7d6c23453b3b4165ab5352323ceed229 P c42f998c7c4947259fd79fdfd9ff7644: Deleting consensus metadata
I20260925 01:17:36.998481 28455 catalog_manager.cc:4841] TS c42f998c7c4947259fd79fdfd9ff7644 (127.20.102.129:36191): tablet 7d6c23453b3b4165ab5352323ceed229 (table test-table [id=b1810258978d414e844162f9a4f954ea]) successfully deleted
I20260925 01:17:37.002264 29055 ts_tablet_manager.cc:1918] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 01:17:37.002419 29055 log.cc:1196] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/7d6c23453b3b4165ab5352323ceed229
I20260925 01:17:37.002899 29055 ts_tablet_manager.cc:1939] T 7d6c23453b3b4165ab5352323ceed229 P 39928671a1cf4d6dbdc6a4e7ff1071ef: Deleting consensus metadata
I20260925 01:17:37.004065 28454 catalog_manager.cc:4841] TS 39928671a1cf4d6dbdc6a4e7ff1071ef (127.20.102.130:44795): tablet 7d6c23453b3b4165ab5352323ceed229 (table test-table [id=b1810258978d414e844162f9a4f954ea]) successfully deleted
Sep 25 01:17:37 dist-test-slave-s12s krb5kdc[28423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.102.129@KRBTEST.COM
I20260925 01:17:37.013617 29026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:36.991125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51342 (local address 127.20.102.129:36191)
0925 01:17:36.991305 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:36.991313 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:36.997810 (+ 6497us) server_negotiation.cc:407] Connection header received
0925 01:17:36.998087 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:36.998092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:36.998169 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:36.998240 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:36.999709 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:37.000646 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:37.002071 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:37.002614 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:37.002753 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:37.007406 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:37.007426 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:37.007449 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:37.007489 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:37.010700 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:37.011402 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:37.011415 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:37.011418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:37.011540 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:37.011952 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:37.011961 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:37.011964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:37.012339 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:37.012668 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:37.013047 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 01:17:37.013292 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:17:37.014295 28671 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:37.182034 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 28540
I20260925 01:17:37.206907 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 28680
Sep 25 01:17:37 dist-test-slave-s12s krb5kdc[28423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.102.130@KRBTEST.COM
I20260925 01:17:37.224309 29033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:37.207815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46296 (local address 127.20.102.130:44795)
0925 01:17:37.208028 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:37.208033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:37.208056 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:17:37.208126 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:37.208132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:37.208248 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:37.208366 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:37.209707 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:37.210508 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:37.211818 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:37.212285 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:37.212444 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:37.217469 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:37.217484 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:37.217503 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:37.217542 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:37.221126 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:37.221981 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:37.221988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:37.221991 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:37.222121 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:37.222585 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:37.222588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:37.222590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:37.223048 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:37.223284 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:37.223730 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 01:17:37.223897 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:17:37.225308 28810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:37.373692 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 28680
I20260925 01:17:37.398744 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 28819
Sep 25 01:17:37 dist-test-slave-s12s krb5kdc[28423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.20.102.131@KRBTEST.COM
I20260925 01:17:37.416061 29089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:37.399678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.20.102.131:36847)
0925 01:17:37.400200 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:37.400204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:37.400221 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:17:37.400278 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:37.400282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:37.400346 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:37.400437 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:37.401859 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:37.402732 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:37.404003 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:37.404445 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:37.404562 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:37.409562 (+ 5000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:37.409578 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:37.409597 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:37.409643 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:37.412596 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:37.413505 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:37.413516 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:37.413519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:37.413693 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:37.414231 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:37.414240 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:37.414245 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:37.414788 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:37.415034 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:37.415439 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 01:17:37.415729 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":309,"threads_started":1}
I20260925 01:17:37.416939 28949 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:37.520430 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 28819
I20260925 01:17:37.539611 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 28436
I20260925 01:17:37.540898 28508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:37.660331 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 28436
2026-09-25T01:17:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5798 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:17:37 dist-test-slave-s12s krb5kdc[29094](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:17:37 dist-test-slave-s12s krb5kdc[29094](info): set up 2 sockets
Sep 25 01:17:37 dist-test-slave-s12s krb5kdc[29094](info): commencing operation
krb5kdc: starting...
W20260925 01:17:38.781821 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:17:38 dist-test-slave-s12s krb5kdc[29094](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:17:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:17:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:17:39.012782 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:40197
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:40197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:39.352145 29107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:39.352422 29107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:39.352547 29107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:39.359957 29107 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:39.360069 29107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:39.360176 29107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:39.360277 29107 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:39.360359 29107 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:17:39.370458 29107 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:40197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:40197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:39.371788 29107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:39.373342 29107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:39.385673 29113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:39.385699 29115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:39.385995 29112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:39.386566 29107 server_base.cc:1034] running on GCE node
I20260925 01:17:39.387410 29107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:39.389036 29107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:39.390281 29107 hybrid_clock.cc:648] HybridClock initialized: now 1790299059390200 us; error 55 us; skew 500 ppm
I20260925 01:17:39.390729 29107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T01:17:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1800 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:17:39 dist-test-slave-s12s krb5kdc[29121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:17:39 dist-test-slave-s12s krb5kdc[29121](info): set up 2 sockets
Sep 25 01:17:39 dist-test-slave-s12s krb5kdc[29121](info): commencing operation
krb5kdc: starting...
W20260925 01:17:40.630189 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:17:40 dist-test-slave-s12s krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:17:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:17:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:17:40.856881 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:33157
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:33157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:41.198055 29134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:41.198407 29134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:41.198583 29134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:41.205719 29134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:41.205873 29134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:41.205981 29134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:41.206085 29134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:41.206152 29134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:17:41.215840 29134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:33157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:33157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:41.217265 29134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:41.218606 29134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:41.230356 29139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:41.230324 29142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:41.230378 29140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:41.231683 29134 server_base.cc:1034] running on GCE node
I20260925 01:17:41.232456 29134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:41.233950 29134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:41.235160 29134 hybrid_clock.cc:648] HybridClock initialized: now 1790299061235106 us; error 51 us; skew 500 ppm
I20260925 01:17:41.235531 29134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:41 dist-test-slave-s12s krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299061, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:41.238741 29134 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:17:41.240291 29134 webserver.cc:460] Webserver started at http://127.20.102.190:33387/ using document root <none> and password file <none>
I20260925 01:17:41.241094 29134 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:41.241225 29134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:41.241559 29134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:41.244616 29134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "d7652f9afe204b84811177adf8be5e87"
format_stamp: "Formatted at 2026-09-25 01:17:41 on dist-test-slave-s12s"
server_key: "98880edc920fb560525b13bc88ae98ff"
server_key_iv: "10c814ec10f8dc2cc46a2e3fa00bf8a5"
server_key_version: "encryptionkey@0"
I20260925 01:17:41.245471 29134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "d7652f9afe204b84811177adf8be5e87"
format_stamp: "Formatted at 2026-09-25 01:17:41 on dist-test-slave-s12s"
server_key: "98880edc920fb560525b13bc88ae98ff"
server_key_iv: "10c814ec10f8dc2cc46a2e3fa00bf8a5"
server_key_version: "encryptionkey@0"
I20260925 01:17:41.250993 29134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:17:41.254527 29149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:41.255924 29134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:17:41.256136 29134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "d7652f9afe204b84811177adf8be5e87"
format_stamp: "Formatted at 2026-09-25 01:17:41 on dist-test-slave-s12s"
server_key: "98880edc920fb560525b13bc88ae98ff"
server_key_iv: "10c814ec10f8dc2cc46a2e3fa00bf8a5"
server_key_version: "encryptionkey@0"
I20260925 01:17:41.256341 29134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:41.284240 29134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:41.288342 29134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:41.288637 29134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:41.307776 29134 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:33157
I20260925 01:17:41.307807 29200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:33157 every 8 connection(s)
I20260925 01:17:41.309226 29134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:17:41.312990 29201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:17:41.319768 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 29134
I20260925 01:17:41.320022 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:17:41.320874 20890 external_mini_cluster.cc:1428] Setting key b2a224f6b8259f4a78713996a284b2d5
I20260925 01:17:41.324752 29201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87: Bootstrap starting.
I20260925 01:17:41.328501 29201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87: Neither blocks nor log segments found. Creating new log.
I20260925 01:17:41.329623 29201 log.cc:824] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:41.332721 29201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87: No bootstrap required, opened a new log
Sep 25 01:17:41 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:41.343585 29201 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7652f9afe204b84811177adf8be5e87" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33157 } }
I20260925 01:17:41.344030 29201 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:17:41.344138 29201 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7652f9afe204b84811177adf8be5e87, State: Initialized, Role: FOLLOWER
I20260925 01:17:41.344823 29201 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7652f9afe204b84811177adf8be5e87" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33157 } }
I20260925 01:17:41.345112 29201 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:17:41.345268 29201 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:17:41.345494 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:41.323030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.20.102.190:33157)
0925 01:17:41.323823 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:41.323835 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:41.323877 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:17:41.325727 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:41.325750 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:41.326168 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:41.326504 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:41.328335 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:41.329786 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:41.331393 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:41.332069 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:41.332202 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:41.336852 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:41.336863 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:41.336883 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:41.336934 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:41.341032 (+ 4098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:41.341685 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:41.341698 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:41.341709 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:41.341936 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:41.342280 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:41.342285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:41.342288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:41.343008 (+ 720us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:41.343290 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:41.343628 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 01:17:41.344099 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":372,"threads_started":1}
I20260925 01:17:41.345460 29201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:17:41.347890 29201 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7652f9afe204b84811177adf8be5e87" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33157 } }
I20260925 01:17:41.348402 29201 leader_election.cc:302] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7652f9afe204b84811177adf8be5e87; no voters:
I20260925 01:17:41.349076 29201 leader_election.cc:288] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:17:41.349212 29206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:17:41.349920 29206 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 1 LEADER]: Becoming Leader. State: Replica: d7652f9afe204b84811177adf8be5e87, State: Running, Role: LEADER
I20260925 01:17:41.350495 29206 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7652f9afe204b84811177adf8be5e87" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33157 } }
I20260925 01:17:41.351079 29201 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:41.353737 29207 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7652f9afe204b84811177adf8be5e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7652f9afe204b84811177adf8be5e87" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33157 } } }
I20260925 01:17:41.354187 29207 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:41.354553 29208 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7652f9afe204b84811177adf8be5e87. Latest consensus state: current_term: 1 leader_uuid: "d7652f9afe204b84811177adf8be5e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7652f9afe204b84811177adf8be5e87" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33157 } } }
I20260925 01:17:41.355316 29208 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:41.355726 29213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:17:41.361084 29213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:17:41.370639 29213 catalog_manager.cc:1347] Generated new cluster ID: 50095fbd8904468fa7f90da1ecfe242a
I20260925 01:17:41.370767 29213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:17:41.377987 29213 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:17:41.379256 29213 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:17:41.390482 29213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87: Generated new TSK 0
I20260925 01:17:41.391458 29213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:17:41.460719 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:33157
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:41.795131 29227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:41.795444 29227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:41.795560 29227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:41.803707 29227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:41.803836 29227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:41.803951 29227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:17:41.814604 29227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:41.815917 29227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:41.817234 29227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:41.831214 29233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:41.831333 29235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:41.831244 29232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:41.832125 29227 server_base.cc:1034] running on GCE node
I20260925 01:17:41.832805 29227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:41.834030 29227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:41.835278 29227 hybrid_clock.cc:648] HybridClock initialized: now 1790299061835235 us; error 64 us; skew 500 ppm
I20260925 01:17:41.835640 29227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:41 dist-test-slave-s12s krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299061, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:41.839955 29227 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:17:41.841687 29227 webserver.cc:460] Webserver started at http://127.20.102.129:37305/ using document root <none> and password file <none>
I20260925 01:17:41.842334 29227 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:41.842438 29227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:41.842705 29227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:41.845697 29227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "10fc2e64cab545d3bfb29373f16bd1ed"
format_stamp: "Formatted at 2026-09-25 01:17:41 on dist-test-slave-s12s"
server_key: "f94d252dbbf2b0774b648358a7f0d1fb"
server_key_iv: "d4f0d6be731f33ad60b5e2ee8589de35"
server_key_version: "encryptionkey@0"
I20260925 01:17:41.846549 29227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "10fc2e64cab545d3bfb29373f16bd1ed"
format_stamp: "Formatted at 2026-09-25 01:17:41 on dist-test-slave-s12s"
server_key: "f94d252dbbf2b0774b648358a7f0d1fb"
server_key_iv: "d4f0d6be731f33ad60b5e2ee8589de35"
server_key_version: "encryptionkey@0"
I20260925 01:17:41.851828 29227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:17:41.855255 29242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:41.856853 29227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:41.857028 29227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "10fc2e64cab545d3bfb29373f16bd1ed"
format_stamp: "Formatted at 2026-09-25 01:17:41 on dist-test-slave-s12s"
server_key: "f94d252dbbf2b0774b648358a7f0d1fb"
server_key_iv: "d4f0d6be731f33ad60b5e2ee8589de35"
server_key_version: "encryptionkey@0"
I20260925 01:17:41.857259 29227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:41.871953 29227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:41.876226 29227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:41.876523 29227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:41.877522 29227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:41.878870 29227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:41.878940 29227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:41.879133 29227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:41.879194 29227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:41.914320 29227 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:33205
I20260925 01:17:41.914367 29354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:33205 every 8 connection(s)
I20260925 01:17:41.915803 29227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:17:41.922355 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 29227
I20260925 01:17:41.922577 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:17:41.923218 20890 external_mini_cluster.cc:1428] Setting key d3670f0791d89a5d614ea9728ddafbd1
Sep 25 01:17:41 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299061, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:41.937412 29357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:41.918554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:49481 (local address 127.20.102.190:33157)
0925 01:17:41.919046 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:41.919052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:41.919883 (+ 831us) server_negotiation.cc:407] Connection header received
0925 01:17:41.921865 (+ 1982us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:41.921876 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:41.921976 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:41.922111 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:41.923837 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:41.924630 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:41.926253 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:41.926667 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:41.926821 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:41.930630 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:41.930636 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:41.930639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:41.930672 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:41.933992 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:41.934773 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:41.934782 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:41.934788 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:41.934943 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:41.935380 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:41.935388 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:41.935394 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:41.935774 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:41.935926 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:41.936633 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 01:17:41.936919 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":260,"threads_started":1}
I20260925 01:17:41.938165 29358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:41.918979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:49481)
0925 01:17:41.919705 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:41.919749 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:41.921579 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:41.922369 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:41.922378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:41.922995 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:41.923661 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:41.923679 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:41.924840 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:41.924845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:41.926096 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:41.926108 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:41.926920 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:41.926931 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:41.927177 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:41.927882 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:41.927906 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:41.930380 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:41.934159 (+ 3779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:41.934166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:41.934180 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:41.934641 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:41.935060 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:41.935063 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:41.935065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:41.935264 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:41.935894 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:41.935900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:41.936243 (+ 343us) client_negotiation.cc:769] Sending connection context
0925 01:17:41.936481 (+ 238us) client_negotiation.cc:240] Negotiation successful
0925 01:17:41.936974 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":267,"threads_started":1}
I20260925 01:17:41.939828 29355 heartbeater.cc:347] Connected to a master server at 127.20.102.190:33157
I20260925 01:17:41.940171 29355 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:41.941123 29355 heartbeater.cc:511] Master 127.20.102.190:33157 requested a full tablet report, sending...
I20260925 01:17:41.943378 29166 ts_manager.cc:194] Registered new tserver with Master: 10fc2e64cab545d3bfb29373f16bd1ed (127.20.102.129:33205)
I20260925 01:17:41.945266 29166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:49481
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:17:41.975692 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:33157
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:42.320071 29361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:42.320350 29361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:42.320533 29361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:42.328085 29361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:42.328200 29361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:42.328434 29361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:17:42.338912 29361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:42.340209 29361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:42.342039 29361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:42.357194 29369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:42.357255 29366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:42.357280 29367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:42.358184 29361 server_base.cc:1034] running on GCE node
I20260925 01:17:42.358819 29361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:42.359882 29361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:42.361099 29361 hybrid_clock.cc:648] HybridClock initialized: now 1790299062361065 us; error 64 us; skew 500 ppm
I20260925 01:17:42.361415 29361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:42 dist-test-slave-s12s krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299062, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:42.364631 29361 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:17:42.366060 29361 webserver.cc:460] Webserver started at http://127.20.102.130:43397/ using document root <none> and password file <none>
I20260925 01:17:42.366737 29361 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:42.366880 29361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:42.367188 29361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:42.370143 29361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "f5c539f71c1248ce96f405cda308ad4a"
format_stamp: "Formatted at 2026-09-25 01:17:42 on dist-test-slave-s12s"
server_key: "9bc4bdbddd8d3cfd56ab68aaa3d7647f"
server_key_iv: "2d571a65771c700afaf00ff954198586"
server_key_version: "encryptionkey@0"
I20260925 01:17:42.371114 29361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "f5c539f71c1248ce96f405cda308ad4a"
format_stamp: "Formatted at 2026-09-25 01:17:42 on dist-test-slave-s12s"
server_key: "9bc4bdbddd8d3cfd56ab68aaa3d7647f"
server_key_iv: "2d571a65771c700afaf00ff954198586"
server_key_version: "encryptionkey@0"
I20260925 01:17:42.376442 29361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:17:42.379971 29376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:42.381325 29361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:42.381507 29361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "f5c539f71c1248ce96f405cda308ad4a"
format_stamp: "Formatted at 2026-09-25 01:17:42 on dist-test-slave-s12s"
server_key: "9bc4bdbddd8d3cfd56ab68aaa3d7647f"
server_key_iv: "2d571a65771c700afaf00ff954198586"
server_key_version: "encryptionkey@0"
I20260925 01:17:42.381721 29361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:42.397408 29361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:42.402267 29361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:42.402607 29361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:42.403625 29361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:42.405143 29361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:42.405257 29361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:42.405445 29361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:42.405524 29361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:42.440439 29361 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:36391
I20260925 01:17:42.440483 29488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:36391 every 8 connection(s)
I20260925 01:17:42.441895 29361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:17:42.449800 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 29361
I20260925 01:17:42.449990 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:17:42.450464 20890 external_mini_cluster.cc:1428] Setting key b1ee9797f7a716d77c81428089fd4e55
Sep 25 01:17:42 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299062, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:42.463758 29491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:42.444710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:39821 (local address 127.20.102.190:33157)
0925 01:17:42.445219 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:42.445227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:42.446139 (+ 912us) server_negotiation.cc:407] Connection header received
0925 01:17:42.447963 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:42.447969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:42.448072 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:42.448195 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:42.449793 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:42.450728 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:42.452586 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:42.453099 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:42.453231 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:42.457158 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:42.457163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:42.457166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:42.457197 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:42.460240 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:42.461235 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:42.461243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:42.461250 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:42.461405 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:42.461979 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:42.462002 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:42.462004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:42.462272 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:42.462443 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:42.463150 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 01:17:42.463379 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":254,"threads_started":1}
I20260925 01:17:42.464952 29492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:42.445185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:39821)
0925 01:17:42.445975 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:42.446019 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:42.447698 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:42.448431 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:42.448439 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:42.449014 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:42.449623 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:42.449641 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:42.450892 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:42.450896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:42.452445 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:42.452457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:42.453326 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:42.453334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:42.453560 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:42.454248 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:42.454269 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:42.456916 (+ 2647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:42.460442 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:42.460449 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:42.460461 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:42.461109 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:42.461563 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:42.461567 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:42.461570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:42.461765 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:42.462417 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:42.462425 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:42.462766 (+ 341us) client_negotiation.cc:769] Sending connection context
0925 01:17:42.463036 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 01:17:42.463592 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":328,"threads_started":1}
I20260925 01:17:42.466585 29489 heartbeater.cc:347] Connected to a master server at 127.20.102.190:33157
I20260925 01:17:42.466907 29489 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:42.467824 29489 heartbeater.cc:511] Master 127.20.102.190:33157 requested a full tablet report, sending...
I20260925 01:17:42.469795 29166 ts_manager.cc:194] Registered new tserver with Master: f5c539f71c1248ce96f405cda308ad4a (127.20.102.130:36391)
I20260925 01:17:42.470809 29166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:39821
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:17:42.502077 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:33157
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:42.825392 29495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:42.825660 29495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:42.825752 29495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:42.832711 29495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:42.832801 29495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:42.832911 29495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:17:42.843822 29495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:42.845151 29495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:42.846451 29495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:42.859591 29501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:42.859658 29503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:42.859591 29500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:42.866070 29495 server_base.cc:1034] running on GCE node
I20260925 01:17:42.866710 29495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:42.867686 29495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:42.868872 29495 hybrid_clock.cc:648] HybridClock initialized: now 1790299062868810 us; error 55 us; skew 500 ppm
I20260925 01:17:42.869182 29495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:42 dist-test-slave-s12s krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299062, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:42.872364 29495 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:17:42.874023 29495 webserver.cc:460] Webserver started at http://127.20.102.131:33951/ using document root <none> and password file <none>
I20260925 01:17:42.874738 29495 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:42.874887 29495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:42.875211 29495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:17:42.878150 29495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "50cddfc3f12543ee83c3f6918076ad50"
format_stamp: "Formatted at 2026-09-25 01:17:42 on dist-test-slave-s12s"
server_key: "d434516defdce417255e1f07c0791f02"
server_key_iv: "07ad478f12525c0e67f4400ab66ee562"
server_key_version: "encryptionkey@0"
I20260925 01:17:42.878939 29495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "50cddfc3f12543ee83c3f6918076ad50"
format_stamp: "Formatted at 2026-09-25 01:17:42 on dist-test-slave-s12s"
server_key: "d434516defdce417255e1f07c0791f02"
server_key_iv: "07ad478f12525c0e67f4400ab66ee562"
server_key_version: "encryptionkey@0"
I20260925 01:17:42.884058 29495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 01:17:42.887646 29510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:42.889007 29495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 01:17:42.889185 29495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "50cddfc3f12543ee83c3f6918076ad50"
format_stamp: "Formatted at 2026-09-25 01:17:42 on dist-test-slave-s12s"
server_key: "d434516defdce417255e1f07c0791f02"
server_key_iv: "07ad478f12525c0e67f4400ab66ee562"
server_key_version: "encryptionkey@0"
I20260925 01:17:42.889396 29495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:42.903705 29495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:42.907956 29495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:42.908257 29495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:42.909267 29495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:42.910537 29495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:42.910650 29495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:42.910800 29495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:42.910876 29495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:42.944866 29495 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:43119
I20260925 01:17:42.944900 29622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:43119 every 8 connection(s)
I20260925 01:17:42.946807 29495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:17:42.948458 29355 heartbeater.cc:503] Master 127.20.102.190:33157 was elected leader, sending a full tablet report...
I20260925 01:17:42.953604 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 29495
I20260925 01:17:42.953779 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:17:42.954334 20890 external_mini_cluster.cc:1428] Setting key fe1e7b47c5f6ce3d0f74352dea533528
Sep 25 01:17:42 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299062, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:17:42.969200 29491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:42.951181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:34929 (local address 127.20.102.190:33157)
0925 01:17:42.951363 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:42.951370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:42.951761 (+ 391us) server_negotiation.cc:407] Connection header received
0925 01:17:42.954644 (+ 2883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:42.954648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:42.954716 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:42.954803 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:42.956797 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:42.957696 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:42.959273 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:42.959756 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:42.959850 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:42.963144 (+ 3294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:42.963149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:42.963152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:42.963182 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:42.965746 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:42.966598 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:42.966602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:42.966604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:42.966717 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:42.967179 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:42.967182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:42.967184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:42.967440 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:42.967574 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:42.968480 (+ 906us) server_negotiation.cc:299] Negotiation successful
0925 01:17:42.968844 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:17:42.970181 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:42.950590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:34929)
0925 01:17:42.951605 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:42.951656 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:42.954340 (+ 2684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:42.955082 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:42.955090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:42.955785 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:42.956601 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:42.956630 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:42.957885 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:42.957889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:42.959152 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:42.959164 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:42.959936 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:42.959941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:42.960145 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:42.960875 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:42.960896 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:42.962955 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:17:42.965916 (+ 2961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:42.965926 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:42.965954 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:42.966474 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:42.966864 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:17:42.966867 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:17:42.966869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:17:42.967073 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:17:42.967639 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:17:42.967645 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:17:42.968048 (+ 403us) client_negotiation.cc:769] Sending connection context
0925 01:17:42.968357 (+ 309us) client_negotiation.cc:240] Negotiation successful
0925 01:17:42.969004 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":657,"threads_started":1}
I20260925 01:17:42.971459 29623 heartbeater.cc:347] Connected to a master server at 127.20.102.190:33157
I20260925 01:17:42.971796 29623 heartbeater.cc:464] Registering TS with master...
I20260925 01:17:42.972537 29623 heartbeater.cc:511] Master 127.20.102.190:33157 requested a full tablet report, sending...
I20260925 01:17:42.973943 29166 ts_manager.cc:194] Registered new tserver with Master: 50cddfc3f12543ee83c3f6918076ad50 (127.20.102.131:43119)
I20260925 01:17:42.974723 29166 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:34929
I20260925 01:17:42.982205 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.129.keytab.
Entry for principal oryx/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.129.keytab.
WARNING: no policy specified for oryx/127.20.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.130.keytab.
Entry for principal oryx/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.130.keytab.
WARNING: no policy specified for oryx/127.20.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.131.keytab.
Entry for principal oryx/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.131.keytab.
I20260925 01:17:43.119330 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 29227
Sep 25 01:17:43 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:17:43.135536 29633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:43.120628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48590 (local address 127.20.102.129:33205)
0925 01:17:43.121356 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:43.121363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:43.121395 (+ 32us) server_negotiation.cc:407] Connection header received
0925 01:17:43.121458 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:43.121465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:43.121688 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:43.121820 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:43.123237 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:43.124232 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:43.125505 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:43.126073 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:43.126169 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:43.130063 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:43.130075 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:43.130089 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:43.130133 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:43.133035 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:43.133757 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:43.133764 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:43.133766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:43.133889 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:43.134223 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:43.134226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:43.134228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:43.134555 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:43.134780 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:43.135022 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 01:17:43.135243 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":377,"threads_started":1}
I20260925 01:17:43.136925 29350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:43.220501 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 29227
I20260925 01:17:43.238669 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 29361
Sep 25 01:17:43 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:17:43.254588 29635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:43.239525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.20.102.130:36391)
0925 01:17:43.240078 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:43.240085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:43.240112 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:17:43.240173 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:43.240180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:43.240391 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:43.240541 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:43.241776 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:43.242792 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:43.243980 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:43.244526 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:43.244627 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:43.248812 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:43.248823 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:43.248837 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:43.248870 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:43.251894 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:43.252476 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:43.252486 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:43.252488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:43.252629 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:43.253129 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:43.253132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:43.253134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:43.253470 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:43.253712 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:43.253972 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 01:17:43.254248 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":280,"threads_started":1}
I20260925 01:17:43.255909 29484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:43.344020 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 29361
I20260925 01:17:43.361227 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 29495
Sep 25 01:17:43 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:17:43.377875 29637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:43.362182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.20.102.131:43119)
0925 01:17:43.362889 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:43.362896 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:43.362925 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:17:43.362985 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:43.362993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:43.363208 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:43.363395 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:43.364801 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:43.365832 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:43.367116 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:43.367708 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:43.367854 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:43.372396 (+ 4542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:43.372406 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:43.372420 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:43.372453 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:43.375279 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:43.375893 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:43.375900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:43.375902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:43.376022 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:43.376332 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:43.376335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:43.376337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:43.376807 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:43.377057 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:43.377287 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 01:17:43.377529 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":380,"threads_started":1}
I20260925 01:17:43.379258 29618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:43.463115 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 29495
I20260925 01:17:43.480273 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 29134
I20260925 01:17:43.481678 29196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:17:43.586192 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 29134
I20260925 01:17:43.609313 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:33157
--webserver_interface=127.20.102.190
--webserver_port=33387
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:33157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:17:43.936192 29640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:43.936483 29640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:43.936617 29640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:43.943982 29640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:17:43.944113 29640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:43.944207 29640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:43.944315 29640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:17:43.944410 29640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:17:43.954463 29640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:33157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:33157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=33387
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:43.955807 29640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:43.957226 29640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:43.969611 29645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:43.969664 29648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:43.969678 29646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:43.970579 29640 server_base.cc:1034] running on GCE node
I20260925 01:17:43.971392 29640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:43.973006 29640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:43.974273 29640 hybrid_clock.cc:648] HybridClock initialized: now 1790299063974221 us; error 55 us; skew 500 ppm
I20260925 01:17:43.974668 29640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:43 dist-test-slave-s12s krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299063, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:43.978052 29640 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:17:43.979621 29640 webserver.cc:460] Webserver started at http://127.20.102.190:33387/ using document root <none> and password file <none>
I20260925 01:17:43.980312 29640 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:43.980445 29640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:43.986073 29640 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:17:43.988893 29655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:43.990237 29640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:43.990378 29640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "d7652f9afe204b84811177adf8be5e87"
format_stamp: "Formatted at 2026-09-25 01:17:41 on dist-test-slave-s12s"
server_key: "98880edc920fb560525b13bc88ae98ff"
server_key_iv: "10c814ec10f8dc2cc46a2e3fa00bf8a5"
server_key_version: "encryptionkey@0"
I20260925 01:17:43.991297 29640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:44.011370 29640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:44.015141 29640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:44.015367 29640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:44.033521 29640 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:33157
I20260925 01:17:44.033540 29706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:33157 every 8 connection(s)
I20260925 01:17:44.035027 29640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:17:44.040910 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 29640
I20260925 01:17:44.041891 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:33205
--local_ip_for_outbound_sockets=127.20.102.129
--tserver_master_addrs=127.20.102.190:33157
--webserver_port=37305
--webserver_interface=127.20.102.129
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.102.129
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:17:44.042448 29707 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:17:44.047556 29707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87: Bootstrap starting.
I20260925 01:17:44.067293 29707 log.cc:824] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87: Log is configured to *not* fsync() on all Append() calls
I20260925 01:17:44.076228 29707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:17:44.076833 29707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87: Bootstrap complete.
I20260925 01:17:44.086005 29707 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7652f9afe204b84811177adf8be5e87" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33157 } }
I20260925 01:17:44.086840 29707 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7652f9afe204b84811177adf8be5e87, State: Initialized, Role: FOLLOWER
I20260925 01:17:44.087538 29707 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7652f9afe204b84811177adf8be5e87" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33157 } }
I20260925 01:17:44.087778 29707 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:17:44.087915 29707 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:17:44.088114 29707 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:17:44.091877 29707 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7652f9afe204b84811177adf8be5e87" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33157 } }
I20260925 01:17:44.092406 29707 leader_election.cc:302] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7652f9afe204b84811177adf8be5e87; no voters:
I20260925 01:17:44.093099 29707 leader_election.cc:288] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 01:17:44.093344 29711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:17:44.094646 29711 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [term 2 LEADER]: Becoming Leader. State: Replica: d7652f9afe204b84811177adf8be5e87, State: Running, Role: LEADER
I20260925 01:17:44.095237 29707 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:17:44.095546 29711 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7652f9afe204b84811177adf8be5e87" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33157 } }
I20260925 01:17:44.102072 29712 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d7652f9afe204b84811177adf8be5e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7652f9afe204b84811177adf8be5e87" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33157 } } }
I20260925 01:17:44.102485 29712 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:44.103075 29713 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7652f9afe204b84811177adf8be5e87. Latest consensus state: current_term: 2 leader_uuid: "d7652f9afe204b84811177adf8be5e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7652f9afe204b84811177adf8be5e87" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33157 } } }
I20260925 01:17:44.103379 29713 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87 [sys.catalog]: This master's current role is: LEADER
I20260925 01:17:44.124960 29718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:17:44.133575 29718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:17:44.135560 29718 catalog_manager.cc:1259] Loaded cluster ID: 50095fbd8904468fa7f90da1ecfe242a
I20260925 01:17:44.135767 29718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:17:44.144825 29718 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:17:44.146880 29718 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d7652f9afe204b84811177adf8be5e87: Loaded TSK: 0
I20260925 01:17:44.149276 29718 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 01:17:44.402650 29709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:44.402999 29709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:44.403187 29709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:44.410692 29709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:44.410854 29709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:44.411041 29709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:17:44.421066 29709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.129.keytab
--principal=oryx/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:33205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=37305
--tserver_master_addrs=127.20.102.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:44.422519 29709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:44.424022 29709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:44.439014 29734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:44.438999 29735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:44.440944 29737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:44.442528 29709 server_base.cc:1034] running on GCE node
I20260925 01:17:44.443153 29709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:44.444106 29709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:44.445331 29709 hybrid_clock.cc:648] HybridClock initialized: now 1790299064445281 us; error 42 us; skew 500 ppm
I20260925 01:17:44.445681 29709 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:44 dist-test-slave-s12s krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299064, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:44.448856 29709 init.cc:377] Logged in from keytab as oryx/127.20.102.129@KRBTEST.COM (short username oryx)
I20260925 01:17:44.450596 29709 webserver.cc:460] Webserver started at http://127.20.102.129:37305/ using document root <none> and password file <none>
I20260925 01:17:44.451314 29709 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:44.451457 29709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:44.457051 29709 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:17:44.459720 29744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:44.461185 29709 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:44.461370 29709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "10fc2e64cab545d3bfb29373f16bd1ed"
format_stamp: "Formatted at 2026-09-25 01:17:41 on dist-test-slave-s12s"
server_key: "f94d252dbbf2b0774b648358a7f0d1fb"
server_key_iv: "d4f0d6be731f33ad60b5e2ee8589de35"
server_key_version: "encryptionkey@0"
I20260925 01:17:44.462281 29709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:44.480469 29709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:44.484885 29709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:44.485154 29709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:44.486152 29709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:44.487519 29709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:44.487639 29709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:44.487776 29709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:44.487852 29709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:44.521236 29709 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:33205
I20260925 01:17:44.521325 29856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:33205 every 8 connection(s)
I20260925 01:17:44.522774 29709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:17:44.527245 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 29709
I20260925 01:17:44.527892 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:36391
--local_ip_for_outbound_sockets=127.20.102.130
--tserver_master_addrs=127.20.102.190:33157
--webserver_port=43397
--webserver_interface=127.20.102.130
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.102.130
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:17:44 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:44 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:44.541599 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:44.525922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:50131)
0925 01:17:44.526745 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:44.526794 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:44.528717 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:44.530124 (+ 1407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:44.530136 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:44.530761 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:44.531303 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:44.531320 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.533021 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.533025 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:44.534262 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:44.534273 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.535386 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.535393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:44.535676 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:44.536398 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:44.536420 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:44.540248 (+ 3828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":409,"threads_started":1}
W20260925 01:17:44.541687 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:44.525842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:50131 (local address 127.20.102.190:33157)
0925 01:17:44.526607 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:44.526623 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:44.526928 (+ 305us) server_negotiation.cc:407] Connection header received
0925 01:17:44.529276 (+ 2348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:44.529288 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:44.529613 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:44.529967 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:44.531515 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.532837 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.534414 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.535126 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.535352 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:44.540640 (+ 5288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:50131: BlockingRecv error: recv got EOF from 127.20.102.129:50131 (error 108)
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":289,"threads_started":1}
W20260925 01:17:44.542496 29857 heartbeater.cc:644] Failed to heartbeat to 127.20.102.190:33157 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.102.190:33157: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Sep 25 01:17:44 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:44 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:44.552668 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:44.543138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:47367)
0925 01:17:44.543380 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:44.543404 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:44.543526 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:44.543841 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:44.543848 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:44.544350 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:44.544590 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:44.544604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.545670 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.545678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:44.547405 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:44.547423 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.548099 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.548108 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:44.548333 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:44.548988 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:44.549016 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:44.552373 (+ 3357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 01:17:44.552861 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:44.543391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:47367 (local address 127.20.102.190:33157)
0925 01:17:44.543570 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:44.543573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:44.543605 (+ 32us) server_negotiation.cc:407] Connection header received
0925 01:17:44.543649 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:44.543652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:44.543704 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:44.543773 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:44.544796 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.545515 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.547563 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.547947 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.548044 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:44.552565 (+ 4521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:47367: BlockingRecv error: recv got EOF from 127.20.102.129:47367 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 01:17:44 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:44 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:44.561697 29860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:44.553442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:37599)
0925 01:17:44.553659 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:44.553678 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:44.553768 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:44.554131 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:44.554135 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:44.554496 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:44.554684 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:44.554694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.555668 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.555674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:44.557201 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:44.557219 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.557863 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.557869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:44.558066 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:44.558652 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:44.558675 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:44.561510 (+ 2835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 01:17:44.561988 29857 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:17:44.562273 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:44.553732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:37599 (local address 127.20.102.190:33157)
0925 01:17:44.553888 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:44.553891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:44.553905 (+ 14us) server_negotiation.cc:407] Connection header received
0925 01:17:44.553943 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:44.553946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:44.554011 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:44.554082 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:44.554816 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.555524 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.557343 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.557724 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.557815 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:44.561732 (+ 3917us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:37599: BlockingRecv error: recv got EOF from 127.20.102.129:37599 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 01:17:44.858232 29861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:44.858493 29861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:44.858583 29861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:44.865669 29861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:44.865760 29861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:44.865878 29861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:17:44.875149 29861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.130.keytab
--principal=oryx/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:36391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=43397
--tserver_master_addrs=127.20.102.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:44.876336 29861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:44.877657 29861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:44.890625 29866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:44.890712 29869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:44.890673 29867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:44.891583 29861 server_base.cc:1034] running on GCE node
I20260925 01:17:44.892086 29861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:44.892938 29861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:44.894179 29861 hybrid_clock.cc:648] HybridClock initialized: now 1790299064894123 us; error 40 us; skew 500 ppm
I20260925 01:17:44.894517 29861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:44 dist-test-slave-s12s krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299064, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:44.898025 29861 init.cc:377] Logged in from keytab as oryx/127.20.102.130@KRBTEST.COM (short username oryx)
I20260925 01:17:44.899358 29861 webserver.cc:460] Webserver started at http://127.20.102.130:43397/ using document root <none> and password file <none>
I20260925 01:17:44.899962 29861 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:44.900079 29861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:44.905566 29861 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:17:44.908322 29876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:44.909744 29861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:17:44.909941 29861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "f5c539f71c1248ce96f405cda308ad4a"
format_stamp: "Formatted at 2026-09-25 01:17:42 on dist-test-slave-s12s"
server_key: "9bc4bdbddd8d3cfd56ab68aaa3d7647f"
server_key_iv: "2d571a65771c700afaf00ff954198586"
server_key_version: "encryptionkey@0"
I20260925 01:17:44.910928 29861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:44.933364 29861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:44.937551 29861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:44.937852 29861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:44.938972 29861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:44.940434 29861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:44.940565 29861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:44.940730 29861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:44.940826 29861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 01:17:44.980926 29861 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:36391
I20260925 01:17:44.980978 29988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:36391 every 8 connection(s)
I20260925 01:17:44.982498 29861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:17:44.989566 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 29861
I20260925 01:17:44.990176 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:43119
--local_ip_for_outbound_sockets=127.20.102.131
--tserver_master_addrs=127.20.102.190:33157
--webserver_port=33951
--webserver_interface=127.20.102.131
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:38987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.20.102.131
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:17:44 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:44 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:17:44.999548 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:44.985201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:48059 (local address 127.20.102.190:33157)
0925 01:17:44.985391 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:44.985396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:44.986572 (+ 1176us) server_negotiation.cc:407] Connection header received
0925 01:17:44.988516 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:44.988521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:44.988619 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:44.988773 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:44.990488 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.991238 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.992959 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.993426 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.993581 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:44.999271 (+ 5690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:48059: BlockingRecv error: recv got EOF from 127.20.102.130:48059 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:17:45.000262 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:44.985622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:48059)
0925 01:17:44.986359 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:44.986402 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:44.988215 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:44.989028 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:44.989037 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:44.989564 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:44.990206 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:44.990225 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.991452 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.991456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:44.992790 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:44.992804 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:44.993682 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:44.993690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:44.993958 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:44.994716 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:44.994747 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:44.999085 (+ 4338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":285,"threads_started":1}
W20260925 01:17:45.000967 29989 heartbeater.cc:644] Failed to heartbeat to 127.20.102.190:33157 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.102.190:33157: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Sep 25 01:17:45 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:45 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:45.010001 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:45.001589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:56579)
0925 01:17:45.001792 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:45.001812 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:45.001905 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:45.002294 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:45.002298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:45.002602 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:45.002734 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:45.002742 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.004138 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.004142 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:45.005178 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:45.005190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.006292 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.006299 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:45.006468 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:45.007085 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:45.007103 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:45.009814 (+ 2711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 01:17:45.010419 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:45.001731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:56579 (local address 127.20.102.190:33157)
0925 01:17:45.001933 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:45.001942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:45.001962 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:17:45.002014 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:45.002021 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:45.002106 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:45.002206 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:45.002880 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.003991 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.005320 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.006107 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.006250 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:45.010105 (+ 3855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:56579: BlockingRecv error: recv got EOF from 127.20.102.130:56579 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 25 01:17:45 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:45 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:45.020179 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:45.010826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:42597)
0925 01:17:45.011025 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:45.011055 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:45.011180 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:45.011494 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:45.011501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:45.011937 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:45.012173 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:45.012188 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.013261 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.013268 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:45.014957 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:45.014973 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.015750 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.015756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:45.015932 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:45.016467 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:45.016489 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:45.019844 (+ 3355us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 01:17:45.020850 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:45.010857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:42597 (local address 127.20.102.190:33157)
0925 01:17:45.011031 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:45.011035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:45.011129 (+ 94us) server_negotiation.cc:407] Connection header received
0925 01:17:45.011286 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:45.011289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:45.011354 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:45.011427 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:45.012355 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.013118 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.015096 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.015623 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.015714 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:45.020557 (+ 4843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:42597: BlockingRecv error: recv got EOF from 127.20.102.130:42597 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 01:17:45.024422 29989 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:17:45.333948 29992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:17:45.334211 29992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:17:45.334304 29992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:17:45.341323 29992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:17:45.341416 29992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:17:45.341589 29992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:17:45.350859 29992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:38987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/krb5kdc/oryx_127.20.102.131.keytab
--principal=oryx/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:43119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=33951
--tserver_master_addrs=127.20.102.190:33157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:17:45.352303 29992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:17:45.353569 29992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:17:45.369323 30000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:45.369429 29997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:17:45.369406 29998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:17:45.370682 29992 server_base.cc:1034] running on GCE node
I20260925 01:17:45.371253 29992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:17:45.372212 29992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:17:45.373461 29992 hybrid_clock.cc:648] HybridClock initialized: now 1790299065373416 us; error 47 us; skew 500 ppm
I20260925 01:17:45.373787 29992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:17:45 dist-test-slave-s12s krb5kdc[29121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299065, etypes {rep=17 tkt=17 ses=17}, oryx/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:17:45.376859 29992 init.cc:377] Logged in from keytab as oryx/127.20.102.131@KRBTEST.COM (short username oryx)
I20260925 01:17:45.378137 29992 webserver.cc:460] Webserver started at http://127.20.102.131:33951/ using document root <none> and password file <none>
I20260925 01:17:45.378831 29992 fs_manager.cc:359] Metadata directory not provided
I20260925 01:17:45.378939 29992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:17:45.384358 29992 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:17:45.387060 30007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:45.388346 29992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 01:17:45.388480 29992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "50cddfc3f12543ee83c3f6918076ad50"
format_stamp: "Formatted at 2026-09-25 01:17:42 on dist-test-slave-s12s"
server_key: "d434516defdce417255e1f07c0791f02"
server_key_iv: "07ad478f12525c0e67f4400ab66ee562"
server_key_version: "encryptionkey@0"
I20260925 01:17:45.389482 29992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:17:45.404270 29992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:17:45.408339 29992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:17:45.408615 29992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:17:45.409720 29992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:17:45.410982 29992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:17:45.411067 29992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:45.411145 29992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:17:45.411198 29992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:17:45.452976 29992 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:43119
I20260925 01:17:45.453037 30119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:43119 every 8 connection(s)
I20260925 01:17:45.454442 29992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:17:45.463631 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 29992
Sep 25 01:17:45 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:45 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:17:45.483469 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:45.457504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:52895 (local address 127.20.102.190:33157)
0925 01:17:45.457736 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:45.457742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:45.459097 (+ 1355us) server_negotiation.cc:407] Connection header received
0925 01:17:45.462269 (+ 3172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:45.462273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:45.462340 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:45.462421 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:45.464794 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.466225 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.469243 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.470083 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.470202 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:45.483080 (+ 12878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:52895: BlockingRecv error: recv got EOF from 127.20.102.131:52895 (error 108)
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":88}
I20260925 01:17:45.485309 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:45.458041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:52895)
0925 01:17:45.458905 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:45.458966 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:45.461937 (+ 2971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:45.462736 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:45.462750 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:45.463631 (+ 881us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:45.464497 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:45.464525 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.466539 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.466558 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:45.469063 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:45.469079 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.470559 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.470568 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:45.470907 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:45.472841 (+ 1934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:45.472878 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:45.483919 (+ 11041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":296,"threads_started":1}
W20260925 01:17:45.486331 30120 heartbeater.cc:644] Failed to heartbeat to 127.20.102.190:33157 (0 consecutive failures): Not authorized: Failed to ping master at 127.20.102.190:33157: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Sep 25 01:17:45 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:45 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:45.495807 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:45.486925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:44841)
0925 01:17:45.487091 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:45.487111 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:45.487210 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:45.487590 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:45.487597 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:45.487992 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:45.488204 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:45.488215 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.489894 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.489902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:45.490819 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:45.490831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.491585 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.491591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:45.491767 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:45.492310 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:45.492327 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:45.495526 (+ 3199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 01:17:45.496029 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:45.487035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:44841 (local address 127.20.102.190:33157)
0925 01:17:45.487240 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:45.487248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:45.487269 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:17:45.487327 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:45.487334 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:45.487418 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:45.487538 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:45.488374 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.489734 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.490969 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.491427 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.491559 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:45.495715 (+ 4156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:44841: BlockingRecv error: recv got EOF from 127.20.102.131:44841 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 01:17:45.500453 30124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:45.464585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.20.102.190:33157)
0925 01:17:45.479869 (+ 15284us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:45.479876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:45.480185 (+ 309us) server_negotiation.cc:407] Connection header received
0925 01:17:45.480901 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:45.480911 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:45.480988 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:45.481101 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:45.482577 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.483414 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.485296 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.486333 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.486478 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:45.490135 (+ 3657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:17:45.490149 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:17:45.490173 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:17:45.490226 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:17:45.495853 (+ 5627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:45.496857 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:45.496869 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:45.496881 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:45.497983 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:17:45.498482 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:17:45.498488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:17:45.498492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:17:45.498989 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:17:45.499337 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0925 01:17:45.499781 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 01:17:45.500166 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":15158,"thread_start_us":15296,"threads_started":1}
Sep 25 01:17:45 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:45 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:45.507045 30122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:45.496935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:41549)
0925 01:17:45.497101 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:45.497117 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:45.497201 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:45.497595 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:45.497598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:45.497846 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:45.498000 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:45.498008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.499776 (+ 1768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.499784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:45.501700 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:45.501720 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.503158 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.503165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:45.503421 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:45.504108 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:45.504124 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:45.506850 (+ 2726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 01:17:45.507354 30120 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:17:45.507431 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:45.496997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:41549 (local address 127.20.102.190:33157)
0925 01:17:45.497246 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:45.497253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:45.497272 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:17:45.497322 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:45.497329 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:45.497448 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:45.497571 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:45.498210 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.499579 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.501903 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.502770 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.502882 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:45.507126 (+ 4244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:41549: BlockingRecv error: recv got EOF from 127.20.102.131:41549 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
Sep 25 01:17:45 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:45 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:45.572153 30125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:45.562650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:37685)
0925 01:17:45.563378 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:45.563395 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:45.563549 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:45.563943 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:45.563949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:45.564452 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:45.564614 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:45.564623 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.565724 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.565730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:45.567023 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:45.567036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.567803 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.567809 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:45.567980 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:45.568669 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:45.568754 (+ 85us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:45.571809 (+ 3055us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":419,"threads_started":1}
W20260925 01:17:45.572269 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:45.562698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:37685 (local address 127.20.102.190:33157)
0925 01:17:45.562891 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:45.562899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:45.563482 (+ 583us) server_negotiation.cc:407] Connection header received
0925 01:17:45.563711 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:45.563717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:45.563795 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:45.563908 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:45.564798 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.565572 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.567161 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:45.567615 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:45.567761 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:45.572039 (+ 4278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:37685: BlockingRecv error: recv got EOF from 127.20.102.129:37685 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 01:17:46 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:46 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:46.033420 30126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:46.025119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:35983)
0925 01:17:46.025669 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:46.025685 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:46.025860 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:46.026243 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:46.026248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:46.026722 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:46.026912 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:46.026921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:46.027943 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:46.027949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:46.028923 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:46.028935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:46.029683 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:46.029688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:46.029854 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:46.030411 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:46.030428 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:46.033080 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":292,"threads_started":1}
W20260925 01:17:46.033583 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:46.025142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:35983 (local address 127.20.102.190:33157)
0925 01:17:46.025362 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:46.025370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:46.025773 (+ 403us) server_negotiation.cc:407] Connection header received
0925 01:17:46.025993 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:46.025999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:46.026084 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:46.026185 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:46.027072 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:46.027794 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:46.029075 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:46.029496 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:46.029629 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:46.033316 (+ 3687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:35983: BlockingRecv error: recv got EOF from 127.20.102.130:35983 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 01:17:46 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:46 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:46.522403 30127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:46.508054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:48529)
0925 01:17:46.508594 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:46.508613 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:46.508790 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:46.509147 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:46.509152 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:46.509625 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:46.509813 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:46.509821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:46.510884 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:46.510896 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:46.511855 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:46.511867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:46.512648 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:46.512654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:46.512884 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:46.518978 (+ 6094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:46.518997 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:46.522070 (+ 3073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":289,"threads_started":1}
W20260925 01:17:46.522588 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:46.508169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:48529 (local address 127.20.102.190:33157)
0925 01:17:46.508412 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:46.508418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:46.508722 (+ 304us) server_negotiation.cc:407] Connection header received
0925 01:17:46.508924 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:46.508928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:46.508987 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:46.509088 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:46.509955 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:46.510735 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:46.512007 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:46.512425 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:46.512589 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:46.522355 (+ 9766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:48529: BlockingRecv error: recv got EOF from 127.20.102.131:48529 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 01:17:46 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:46 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:17:46.581691 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:46.573316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:54127 (local address 127.20.102.190:33157)
0925 01:17:46.573564 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:46.573571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:46.573872 (+ 301us) server_negotiation.cc:407] Connection header received
0925 01:17:46.574057 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:46.574064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:46.574169 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:46.574277 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:46.575273 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:46.576086 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:46.577304 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:46.577721 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:46.577809 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:46.581467 (+ 3658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:54127: BlockingRecv error: recv got EOF from 127.20.102.129:54127 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 01:17:46.581701 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:46.573240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:54127)
0925 01:17:46.573751 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:46.573767 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:46.573929 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:46.574334 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:46.574339 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:46.574841 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:46.575098 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:46.575113 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:46.576225 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:46.576230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:46.577191 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:46.577204 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:46.577852 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:46.577857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:46.577973 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:46.578544 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:46.578561 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:46.581286 (+ 2725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":277,"threads_started":1}
Sep 25 01:17:47 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:47 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:47.044250 30129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:47.034590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:47431)
0925 01:17:47.035322 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:47.035338 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:47.035453 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:47.035783 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:47.035786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:47.036188 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:47.036472 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:47.036490 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:47.038004 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:47.038011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:47.039479 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:47.039504 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:47.040199 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:47.040206 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:47.040375 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:47.041129 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:47.041152 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:47.044023 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":468,"threads_started":1}
W20260925 01:17:47.044407 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:47.034646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:47431 (local address 127.20.102.190:33157)
0925 01:17:47.034883 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:47.034893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:47.035420 (+ 527us) server_negotiation.cc:407] Connection header received
0925 01:17:47.035567 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:47.035572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:47.035648 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:47.035757 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:47.036748 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:47.037884 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:47.039630 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:47.040052 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:47.040141 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:47.044209 (+ 4068us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:47431: BlockingRecv error: recv got EOF from 127.20.102.130:47431 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 01:17:47 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:47 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:47.531682 30130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:47.523317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:55497)
0925 01:17:47.523875 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:47.523896 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:47.524053 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:47.524396 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:47.524400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:47.524878 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:47.525062 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:47.525072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:47.526076 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:47.526084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:47.527117 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:47.527130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:47.527881 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:47.527888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:47.528068 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:47.528612 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:47.528630 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:47.531438 (+ 2808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":308,"threads_started":1}
W20260925 01:17:47.531888 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:47.523467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:55497 (local address 127.20.102.190:33157)
0925 01:17:47.523716 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:47.523724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:47.523978 (+ 254us) server_negotiation.cc:407] Connection header received
0925 01:17:47.524181 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:47.524186 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:47.524262 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:47.524358 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:47.525203 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:47.525928 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:47.527262 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:47.527716 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:47.527848 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:47.531625 (+ 3777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:55497: BlockingRecv error: recv got EOF from 127.20.102.131:55497 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 01:17:47 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:47 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:47.590677 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:47.582523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:56179)
0925 01:17:47.583039 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:47.583056 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:47.583168 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:47.583445 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:47.583448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:47.583808 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:47.583983 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:47.583992 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:47.585145 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:47.585149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:47.586045 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:47.586057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:47.586749 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:47.586753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:47.586892 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:47.587537 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:47.587581 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:47.590392 (+ 2811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":286,"threads_started":1}
W20260925 01:17:47.590936 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:47.582632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:56179 (local address 127.20.102.190:33157)
0925 01:17:47.582791 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:47.582796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:47.583145 (+ 349us) server_negotiation.cc:407] Connection header received
0925 01:17:47.583264 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:47.583267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:47.583329 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:47.583405 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:47.584132 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:47.585007 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:47.586186 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:47.586614 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:47.586715 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:47.590586 (+ 3871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:56179: BlockingRecv error: recv got EOF from 127.20.102.129:56179 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 01:17:48 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:48 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:48.053895 30132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:48.045225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:58977)
0925 01:17:48.045725 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:48.045743 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:48.045869 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:48.046168 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:48.046171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:48.046507 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:48.046671 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:48.046679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:48.047766 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:48.047771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:48.048858 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:48.048875 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:48.049550 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:48.049555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:48.049690 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:48.050316 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:48.050335 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:48.053471 (+ 3136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":290,"threads_started":1}
W20260925 01:17:48.053953 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:48.045312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:58977 (local address 127.20.102.190:33157)
0925 01:17:48.045481 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:48.045486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:48.045831 (+ 345us) server_negotiation.cc:407] Connection header received
0925 01:17:48.045974 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:48.045978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:48.046043 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:48.046124 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:48.046802 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:48.047597 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:48.049003 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:48.049405 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:48.049546 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:48.053690 (+ 4144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:58977: BlockingRecv error: recv got EOF from 127.20.102.130:58977 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 01:17:48 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:48 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:48.540482 30133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:48.532603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:52599)
0925 01:17:48.533098 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:48.533120 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:48.533232 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:48.533641 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:48.533646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:48.533998 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:48.534167 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:48.534177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:48.535153 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:48.535156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:48.536067 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:48.536078 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:48.536794 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:48.536799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:48.536921 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:48.537491 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:48.537509 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:48.540226 (+ 2717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":285,"threads_started":1}
W20260925 01:17:48.540843 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:48.532804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:52599 (local address 127.20.102.190:33157)
0925 01:17:48.533027 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:48.533033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:48.533196 (+ 163us) server_negotiation.cc:407] Connection header received
0925 01:17:48.533330 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:48.533333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:48.533449 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:48.533603 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:48.534317 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:48.535035 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:48.536200 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:48.536625 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:48.536795 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:48.540592 (+ 3797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:52599: BlockingRecv error: recv got EOF from 127.20.102.131:52599 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 01:17:48 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:48 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:48.600003 30134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:48.591647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:43105)
0925 01:17:48.592158 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:48.592174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:48.592332 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:48.592758 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:48.592763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:48.593162 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:48.593308 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:48.593317 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:48.594308 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:48.594314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:48.595375 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:48.595387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:48.596188 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:48.596194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:48.596370 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:48.597057 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:48.597075 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:48.599794 (+ 2719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":270,"threads_started":1}
W20260925 01:17:48.600404 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:48.591734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:43105 (local address 127.20.102.190:33157)
0925 01:17:48.591967 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:48.591985 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:48.592257 (+ 272us) server_negotiation.cc:407] Connection header received
0925 01:17:48.592460 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:48.592465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:48.592552 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:48.592721 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:48.593452 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:48.594175 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:48.595531 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:48.596038 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:48.596188 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:48.600014 (+ 3826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:43105: BlockingRecv error: recv got EOF from 127.20.102.129:43105 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 01:17:49 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:49 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:49.063560 30135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:49.054883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:33815)
0925 01:17:49.055392 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:49.055408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:49.055583 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:49.055947 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:49.055952 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:49.056359 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:49.056539 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:49.056548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:49.057663 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:49.057670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:49.058667 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:49.058679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:49.059389 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:49.059394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:49.059574 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:49.060287 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:49.060308 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:49.063211 (+ 2903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":269,"threads_started":1}
W20260925 01:17:49.063637 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:49.054954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:33815 (local address 127.20.102.190:33157)
0925 01:17:49.055221 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:49.055228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:49.055505 (+ 277us) server_negotiation.cc:407] Connection header received
0925 01:17:49.055725 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:49.055730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:49.055802 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:49.055905 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:49.056715 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:49.057512 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:49.058812 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:49.059242 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:49.059380 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:49.063411 (+ 4031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:33815: BlockingRecv error: recv got EOF from 127.20.102.130:33815 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 25 01:17:49 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:49 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:49.551278 30136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:49.541530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:41287)
0925 01:17:49.542097 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:49.542119 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:49.542299 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:49.542706 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:49.542712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:49.543147 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:49.543313 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:49.543324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:49.544520 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:49.544529 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:49.545676 (+ 1147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:49.545690 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:49.546698 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:49.546706 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:49.546902 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:49.547748 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:49.547774 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:49.550948 (+ 3174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":304,"threads_started":1}
W20260925 01:17:49.551426 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:49.541679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:41287 (local address 127.20.102.190:33157)
0925 01:17:49.541950 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:49.541958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:49.542219 (+ 261us) server_negotiation.cc:407] Connection header received
0925 01:17:49.542460 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:49.542467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:49.542549 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:49.542666 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:49.543487 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:49.544356 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:49.545841 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:49.546479 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:49.546632 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:49.551171 (+ 4539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:41287: BlockingRecv error: recv got EOF from 127.20.102.131:41287 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 01:17:49 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:49 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:49.609603 30137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:49.601025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:37277)
0925 01:17:49.601583 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:49.601599 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:49.601710 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:49.602011 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:49.602014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:49.602453 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:49.602605 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:49.602614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:49.603726 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:49.603729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:49.604643 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:49.604653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:49.605386 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:49.605391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:49.605508 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:49.606095 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:49.606112 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:49.609255 (+ 3143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":297,"threads_started":1}
W20260925 01:17:49.609699 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:49.601099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:37277 (local address 127.20.102.190:33157)
0925 01:17:49.601289 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:49.601293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:49.601676 (+ 383us) server_negotiation.cc:407] Connection header received
0925 01:17:49.601818 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:49.601822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:49.601887 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:49.601986 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:49.602750 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:49.603578 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:49.604819 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:49.605242 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:49.605351 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:49.609469 (+ 4118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:37277: BlockingRecv error: recv got EOF from 127.20.102.129:37277 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 01:17:50 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:50 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:50.073930 30138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:50.064880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:34003)
0925 01:17:50.065422 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:50.065440 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:50.065565 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:50.065870 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:50.065873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:50.066333 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:50.066493 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:50.066502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:50.067960 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:50.067964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:50.069086 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:50.069099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:50.069894 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:50.069898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:50.070014 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:50.070657 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:50.070674 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:50.073675 (+ 3001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":291,"threads_started":1}
W20260925 01:17:50.074169 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:50.065135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:34003 (local address 127.20.102.190:33157)
0925 01:17:50.065362 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:50.065370 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:50.065539 (+ 169us) server_negotiation.cc:407] Connection header received
0925 01:17:50.065675 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:50.065679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:50.065743 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:50.065826 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:50.066685 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:50.067818 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:50.069261 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:50.069752 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:50.069869 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:50.073879 (+ 4010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:34003: BlockingRecv error: recv got EOF from 127.20.102.130:34003 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 01:17:50 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:50 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:50.560889 30139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:50.552415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:40391)
0925 01:17:50.553015 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:50.553035 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:50.553147 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:50.553493 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:50.553496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:50.553921 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:50.554076 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:50.554085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:50.555090 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:50.555094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:50.556021 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:50.556032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:50.556706 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:50.556711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:50.556840 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:50.557400 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:50.557417 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:50.560479 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":340,"threads_started":1}
W20260925 01:17:50.561169 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:50.552470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:40391 (local address 127.20.102.190:33157)
0925 01:17:50.552737 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:50.552748 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:50.553142 (+ 394us) server_negotiation.cc:407] Connection header received
0925 01:17:50.553277 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:50.553281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:50.553361 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:50.553451 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:50.554232 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:50.554967 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:50.556147 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:50.556549 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:50.556641 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:50.560907 (+ 4266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:40391: BlockingRecv error: recv got EOF from 127.20.102.131:40391 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 01:17:50 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:50 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:50.619335 30140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:50.610609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:44595)
0925 01:17:50.611175 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:50.611194 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:50.611335 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:50.611665 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:50.611669 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:50.612109 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:50.612293 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:50.612303 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:50.613372 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:50.613376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:50.614390 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:50.614404 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:50.615230 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:50.615235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:50.615371 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:50.616114 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:50.616134 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:50.619091 (+ 2957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":317,"threads_started":1}
W20260925 01:17:50.619625 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:50.610709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:44595 (local address 127.20.102.190:33157)
0925 01:17:50.610885 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:50.610889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:50.611272 (+ 383us) server_negotiation.cc:407] Connection header received
0925 01:17:50.611439 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:50.611443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:50.611504 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:50.611609 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:50.612422 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:50.613218 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:50.614583 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:50.614989 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:50.615141 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:50.619287 (+ 4146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:44595: BlockingRecv error: recv got EOF from 127.20.102.129:44595 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 01:17:51 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:51 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:51.083257 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:51.074874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:43565)
0925 01:17:51.075442 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:51.075458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:51.075572 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:51.075883 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:51.075886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:51.076271 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:51.076435 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:51.076443 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:51.077467 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:51.077471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:51.078357 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:51.078367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:51.079086 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:51.079090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:51.079208 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:51.079769 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:51.079785 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:51.083039 (+ 3254us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":327,"threads_started":1}
W20260925 01:17:51.083567 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:51.074957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:43565 (local address 127.20.102.190:33157)
0925 01:17:51.075139 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:51.075144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:51.075564 (+ 420us) server_negotiation.cc:407] Connection header received
0925 01:17:51.075683 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:51.075686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:51.075751 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:51.075836 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:51.076564 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:51.077333 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:51.078487 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:51.078938 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:51.079038 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:51.083249 (+ 4211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:43565: BlockingRecv error: recv got EOF from 127.20.102.130:43565 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 01:17:51 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:51 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:51.574914 30142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:51.562006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:60767)
0925 01:17:51.562711 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:51.562732 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:51.562882 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:51.563392 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:51.563397 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:51.563833 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:51.564086 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:51.564097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:51.565238 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:51.565253 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:51.566159 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:51.566171 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:51.567087 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:51.567091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:51.567219 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:51.571642 (+ 4423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:51.571661 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:51.574611 (+ 2950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":415,"threads_started":1}
W20260925 01:17:51.575116 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:51.562135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:60767 (local address 127.20.102.190:33157)
0925 01:17:51.562360 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:51.562367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:51.562797 (+ 430us) server_negotiation.cc:407] Connection header received
0925 01:17:51.563041 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:51.563046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:51.563128 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:51.563237 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:51.564220 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:51.565063 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:51.566359 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:51.566807 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:51.566881 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:51.574805 (+ 7924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:60767: BlockingRecv error: recv got EOF from 127.20.102.131:60767 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 01:17:51 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:51 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:51.629276 30143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:51.620377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:40479)
0925 01:17:51.621075 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:51.621090 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:51.621209 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:51.621531 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:51.621534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:51.621937 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:51.622141 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:51.622154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:51.623617 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:51.623623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:51.624655 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:51.624668 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:51.625398 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:51.625402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:51.625547 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:51.626121 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:51.626138 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:51.629030 (+ 2892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":390,"threads_started":1}
W20260925 01:17:51.629549 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:51.620462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:40479 (local address 127.20.102.190:33157)
0925 01:17:51.620723 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:51.620729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:51.621176 (+ 447us) server_negotiation.cc:407] Connection header received
0925 01:17:51.621309 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:51.621312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:51.621377 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:51.621457 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:51.622313 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:51.623448 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:51.624841 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:51.625239 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:51.625332 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:51.629270 (+ 3938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:40479: BlockingRecv error: recv got EOF from 127.20.102.129:40479 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 25 01:17:52 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:52 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:52.093201 30144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:52.084143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:46685)
0925 01:17:52.084964 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:52.084981 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:52.085153 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:52.085518 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:52.085523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:52.085975 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:52.086152 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:52.086161 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:52.087204 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:52.087209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:52.088288 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:52.088300 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:52.089162 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:52.089168 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:52.089367 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:52.090009 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:52.090029 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:52.092813 (+ 2784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":515,"threads_started":1}
W20260925 01:17:52.093319 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:52.084267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:46685 (local address 127.20.102.190:33157)
0925 01:17:52.084537 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:52.084547 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:52.085076 (+ 529us) server_negotiation.cc:407] Connection header received
0925 01:17:52.085294 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:52.085298 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:52.085376 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:52.085482 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:52.086306 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:52.087060 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:52.088430 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:52.088992 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:52.089133 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:52.093031 (+ 3898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:46685: BlockingRecv error: recv got EOF from 127.20.102.130:46685 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 25 01:17:52 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:52 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:52.585176 30145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:52.575866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:39901)
0925 01:17:52.576394 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:52.576414 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:52.576561 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:52.576905 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:52.576910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:52.577363 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:52.577723 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:52.577733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:52.578775 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:52.578782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:52.579765 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:52.579777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:52.580827 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:52.580831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:52.581037 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:52.581715 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:52.581742 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:52.584811 (+ 3069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":255,"threads_started":1}
W20260925 01:17:52.585268 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:52.575982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:39901 (local address 127.20.102.190:33157)
0925 01:17:52.576219 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:52.576226 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:52.576516 (+ 290us) server_negotiation.cc:407] Connection header received
0925 01:17:52.576662 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:52.576665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:52.576758 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:52.576871 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:52.577887 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:52.578635 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:52.579992 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:52.580637 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:52.580812 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:52.585016 (+ 4204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:39901: BlockingRecv error: recv got EOF from 127.20.102.131:39901 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 01:17:52 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:52 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:52.638871 30146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:52.630345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:54159)
0925 01:17:52.631098 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:52.631123 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:52.631269 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:52.631650 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:52.631655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:52.632093 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:52.632247 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:52.632255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:52.633284 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:52.633288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:52.634226 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:52.634237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:52.634964 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:52.634968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:52.635088 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:52.635674 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:52.635691 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:52.638537 (+ 2846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":441,"threads_started":1}
W20260925 01:17:52.639128 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:52.630617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:54159 (local address 127.20.102.190:33157)
0925 01:17:52.630828 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:52.630834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:52.631188 (+ 354us) server_negotiation.cc:407] Connection header received
0925 01:17:52.631399 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:52.631407 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:52.631486 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:52.631575 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:52.632366 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:52.633169 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:52.634342 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:52.634814 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:52.634906 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:52.638745 (+ 3839us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:54159: BlockingRecv error: recv got EOF from 127.20.102.129:54159 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 01:17:53 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:53 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:53.103147 30147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:53.094324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:49505)
0925 01:17:53.094817 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:53.094834 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:53.094948 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:53.095304 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:53.095311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:53.095742 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:53.095887 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:53.095895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:53.097464 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:53.097469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:53.098620 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:53.098633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:53.099332 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:53.099336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:53.099440 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:53.100024 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:53.100041 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:53.102918 (+ 2877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":262,"threads_started":1}
W20260925 01:17:53.103318 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:53.094408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:49505 (local address 127.20.102.190:33157)
0925 01:17:53.094618 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:53.094624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:53.094893 (+ 269us) server_negotiation.cc:407] Connection header received
0925 01:17:53.095042 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:53.095046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:53.095119 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:53.095206 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:53.096061 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:53.097325 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:53.098750 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:53.099180 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:53.099278 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:53.103084 (+ 3806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:49505: BlockingRecv error: recv got EOF from 127.20.102.130:49505 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 01:17:53 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:53 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:53.595921 30148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:53.586255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:39167)
0925 01:17:53.586834 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:53.586855 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:53.586979 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:53.587301 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:53.587305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:53.587714 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:53.587878 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:53.587887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:53.589065 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:53.589072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:53.590333 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:53.590347 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:53.591283 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:53.591289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:53.591468 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:53.592199 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:53.592219 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:53.595546 (+ 3327us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":305,"threads_started":1}
W20260925 01:17:53.595948 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:53.586338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:39167 (local address 127.20.102.190:33157)
0925 01:17:53.586550 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:53.586555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:53.586933 (+ 378us) server_negotiation.cc:407] Connection header received
0925 01:17:53.587091 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:53.587094 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:53.587172 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:53.587257 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:53.588002 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:53.588853 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:53.590488 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:53.591047 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:53.591234 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:53.595758 (+ 4524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:39167: BlockingRecv error: recv got EOF from 127.20.102.131:39167 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 01:17:53 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:53 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:53.648289 30149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:53.639835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:44751)
0925 01:17:53.640359 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:53.640375 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:53.640487 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:53.640770 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:53.640774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:53.641172 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:53.641315 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:53.641324 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:53.642335 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:53.642338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:53.643284 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:53.643298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:53.643995 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:53.644000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:53.644138 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:53.644908 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:53.644929 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:53.648013 (+ 3084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":273,"threads_started":1}
W20260925 01:17:53.648427 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:53.639873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:44751 (local address 127.20.102.190:33157)
0925 01:17:53.640071 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:53.640075 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:53.640448 (+ 373us) server_negotiation.cc:407] Connection header received
0925 01:17:53.640595 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:53.640599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:53.640659 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:53.640753 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:53.641437 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:53.642206 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:53.643407 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:53.643813 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:53.643916 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:53.648211 (+ 4295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:44751: BlockingRecv error: recv got EOF from 127.20.102.129:44751 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 01:17:54 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:54 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:54.113101 30150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:54.104104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:47365)
0925 01:17:54.104778 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:54.104794 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:54.104946 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:54.105259 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:54.105262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:54.105717 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:54.105885 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:54.105894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:54.106885 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:54.106889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:54.107816 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:54.107827 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:54.108563 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:54.108568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:54.108783 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:54.109474 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:54.109494 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:54.112584 (+ 3090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":319,"threads_started":1}
W20260925 01:17:54.113282 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:54.104154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:47365 (local address 127.20.102.190:33157)
0925 01:17:54.104328 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:54.104333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:54.104934 (+ 601us) server_negotiation.cc:407] Connection header received
0925 01:17:54.105057 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:54.105061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:54.105125 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:54.105210 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:54.106022 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:54.106756 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:54.107944 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:54.108377 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:54.108471 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:54.113049 (+ 4578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:47365: BlockingRecv error: recv got EOF from 127.20.102.130:47365 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 01:17:54 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:54 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:54.605791 30151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:54.596929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:48495)
0925 01:17:54.597440 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:54.597460 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:54.597620 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:54.597967 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:54.597972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:54.598399 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:54.598566 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:54.598576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:54.599579 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:54.599587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:54.600546 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:54.600559 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:54.601311 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:54.601317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:54.601513 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:54.602223 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:54.602243 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:54.605515 (+ 3272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":287,"threads_started":1}
W20260925 01:17:54.605919 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:54.596961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:48495 (local address 127.20.102.190:33157)
0925 01:17:54.597183 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:54.597191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:54.597560 (+ 369us) server_negotiation.cc:407] Connection header received
0925 01:17:54.597750 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:54.597755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:54.597826 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:54.597931 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:54.598717 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:54.599440 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:54.600721 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:54.601148 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:54.601296 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:54.605696 (+ 4400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:48495: BlockingRecv error: recv got EOF from 127.20.102.131:48495 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 01:17:54 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:54 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:54.658757 30152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:54.649287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:48333)
0925 01:17:54.649881 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:54.649903 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:54.650050 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:54.650424 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:54.650429 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:54.650984 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:54.651201 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:54.651217 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:54.652643 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:54.652647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:54.653661 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:54.653676 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:54.654778 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:54.654783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:54.654894 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:54.655479 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:54.655495 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:54.658406 (+ 2911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":301,"threads_started":1}
W20260925 01:17:54.658838 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:54.649373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:48333 (local address 127.20.102.190:33157)
0925 01:17:54.649627 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:54.649635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:54.650013 (+ 378us) server_negotiation.cc:407] Connection header received
0925 01:17:54.650183 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:54.650188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:54.650275 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:54.650353 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:54.651412 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:54.652493 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:54.653824 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:54.654622 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:54.654726 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:54.658586 (+ 3860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:48333: BlockingRecv error: recv got EOF from 127.20.102.129:48333 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 25 01:17:55 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:55 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:55.123133 30153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:55.114174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:47147)
0925 01:17:55.114920 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:55.114943 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:55.115167 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:55.115664 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:55.115668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:55.116236 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:55.116380 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:55.116389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:55.117450 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:55.117453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:55.118389 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:55.118399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:55.119078 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:55.119081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:55.119192 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:55.119808 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:55.119826 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:55.122779 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":483,"threads_started":1}
W20260925 01:17:55.123270 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:55.114274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:47147 (local address 127.20.102.190:33157)
0925 01:17:55.114462 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:55.114468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:55.115032 (+ 564us) server_negotiation.cc:407] Connection header received
0925 01:17:55.115427 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:55.115434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:55.115521 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:55.115636 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:55.116521 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:55.117327 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:55.118528 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:55.118932 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:55.119035 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:55.123000 (+ 3965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:47147: BlockingRecv error: recv got EOF from 127.20.102.130:47147 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 01:17:55 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:55 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:55.615130 30154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:55.606698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:54819)
0925 01:17:55.607239 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:55.607259 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:55.607438 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:55.607807 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:55.607810 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:55.608165 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:55.608339 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:55.608347 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:55.609667 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:55.609671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:55.610616 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:55.610627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:55.611281 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:55.611285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:55.611397 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:55.611943 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:55.611959 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:55.614820 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":280,"threads_started":1}
W20260925 01:17:55.615279 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:55.606877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:54819 (local address 127.20.102.190:33157)
0925 01:17:55.607100 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:55.607105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:55.607335 (+ 230us) server_negotiation.cc:407] Connection header received
0925 01:17:55.607625 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:55.607628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:55.607692 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:55.607769 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:55.608533 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:55.609535 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:55.610744 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:55.611153 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:55.611245 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:55.615014 (+ 3769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:54819: BlockingRecv error: recv got EOF from 127.20.102.131:54819 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 01:17:55 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:55 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:55.667971 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:55.659606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:46667)
0925 01:17:55.660072 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:55.660087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:55.660197 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:55.660495 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:55.660497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:55.660942 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:55.661127 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:55.661137 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:55.662170 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:55.662174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:55.663310 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:55.663324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:55.664063 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:55.664069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:55.664240 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:55.664861 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:55.664877 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:55.667727 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":246,"threads_started":1}
W20260925 01:17:55.668118 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:55.659701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:46667 (local address 127.20.102.190:33157)
0925 01:17:55.659954 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:55.659962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:55.660168 (+ 206us) server_negotiation.cc:407] Connection header received
0925 01:17:55.660311 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:55.660314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:55.660373 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:55.660448 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:55.661256 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:55.662010 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:55.663441 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:55.663904 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:55.664032 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:55.667900 (+ 3868us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:46667: BlockingRecv error: recv got EOF from 127.20.102.129:46667 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 25 01:17:56 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:56 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:56.132378 30156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:56.124195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:42105)
0925 01:17:56.124736 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:56.124753 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:56.124913 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:56.125315 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:56.125320 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:56.125789 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:56.126032 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:56.126042 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:56.127067 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:56.127071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:56.127949 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:56.127960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:56.128616 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:56.128620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:56.128806 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:56.129421 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:56.129438 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:56.132143 (+ 2705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":264,"threads_started":1}
W20260925 01:17:56.132581 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:56.124253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:42105 (local address 127.20.102.190:33157)
0925 01:17:56.124461 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:56.124468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:56.124853 (+ 385us) server_negotiation.cc:407] Connection header received
0925 01:17:56.125067 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:56.125073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:56.125150 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:56.125237 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:56.126206 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:56.126946 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:56.128067 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:56.128465 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:56.128565 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:56.132342 (+ 3777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:42105: BlockingRecv error: recv got EOF from 127.20.102.130:42105 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 01:17:56 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:56 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:56.628605 30157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:56.616249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:55621)
0925 01:17:56.616753 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:56.616773 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:56.616932 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:56.617261 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:56.617266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:56.617715 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:56.617888 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:56.617897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:56.619004 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:56.619011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:56.619905 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:56.619916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:56.620604 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:56.620610 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:56.620822 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:56.625246 (+ 4424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:56.625266 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:56.628325 (+ 3059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":262,"threads_started":1}
W20260925 01:17:56.628850 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:56.616323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:55621 (local address 127.20.102.190:33157)
0925 01:17:56.616598 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:56.616604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:56.616863 (+ 259us) server_negotiation.cc:407] Connection header received
0925 01:17:56.617065 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:56.617069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:56.617130 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:56.617229 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:56.618095 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:56.618855 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:56.620053 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:56.620444 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:56.620590 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:56.628567 (+ 7977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:55621: BlockingRecv error: recv got EOF from 127.20.102.131:55621 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 25 01:17:56 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:56 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:56.677852 30158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:56.668904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:50213)
0925 01:17:56.669451 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:56.669467 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:56.669611 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:56.669909 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:56.669912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:56.670395 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:56.670616 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:56.670628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:56.672134 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:56.672140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:56.673238 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:56.673254 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:56.673958 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:56.673962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:56.674090 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:56.674650 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:56.674668 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:56.677554 (+ 2886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":315,"threads_started":1}
W20260925 01:17:56.678088 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:56.668981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:50213 (local address 127.20.102.190:33157)
0925 01:17:56.669168 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:56.669173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:56.669591 (+ 418us) server_negotiation.cc:407] Connection header received
0925 01:17:56.669716 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:56.669719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:56.669782 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:56.669865 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:56.670784 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:56.671980 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:56.673365 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:56.673836 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:56.673930 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:56.677795 (+ 3865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:50213: BlockingRecv error: recv got EOF from 127.20.102.129:50213 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 01:17:57 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:57 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:57.141526 30159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:57.133359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:41359)
0925 01:17:57.133926 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:57.133941 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:57.134058 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:57.134351 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:57.134354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:57.134749 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:57.134903 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:57.134911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:57.135848 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:57.135852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:57.136818 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:57.136831 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:57.137462 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:57.137466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:57.137617 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:57.138219 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:57.138237 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:57.141229 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":329,"threads_started":1}
W20260925 01:17:57.141774 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:57.133455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:41359 (local address 127.20.102.190:33157)
0925 01:17:57.133659 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:57.133664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:57.134012 (+ 348us) server_negotiation.cc:407] Connection header received
0925 01:17:57.134161 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:57.134164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:57.134223 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:57.134303 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:57.135025 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:57.135720 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:57.136937 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:57.137326 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:57.137431 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:57.141497 (+ 4066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:41359: BlockingRecv error: recv got EOF from 127.20.102.130:41359 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 01:17:57 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:57 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:57.638253 30160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:57.629680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:37315)
0925 01:17:57.630196 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:57.630216 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:57.630369 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:57.630731 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:57.630735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:57.631171 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:57.631331 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:57.631340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:57.632396 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:57.632401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:57.633529 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:57.633542 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:57.634465 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:57.634471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:57.634672 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:57.635306 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:57.635324 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:57.638026 (+ 2702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":308,"threads_started":1}
W20260925 01:17:57.638438 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:57.629711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:37315 (local address 127.20.102.190:33157)
0925 01:17:57.629932 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:57.629941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:57.630305 (+ 364us) server_negotiation.cc:407] Connection header received
0925 01:17:57.630509 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:57.630513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:57.630574 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:57.630683 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:57.631477 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:57.632235 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:57.633712 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:57.634308 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:57.634441 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:57.638209 (+ 3768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:37315: BlockingRecv error: recv got EOF from 127.20.102.131:37315 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 01:17:57 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:57 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:57.687132 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:57.678828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:39235)
0925 01:17:57.679354 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:57.679370 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:57.679481 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:57.679768 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:57.679772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:57.680226 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:57.680499 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:57.680513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:57.681726 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:57.681731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:57.682690 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:57.682701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:57.683413 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:57.683418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:57.683587 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:57.684093 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:57.684109 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:57.686880 (+ 2771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":263,"threads_started":1}
W20260925 01:17:57.687417 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:57.678888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:39235 (local address 127.20.102.190:33157)
0925 01:17:57.679079 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:57.679083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:57.679441 (+ 358us) server_negotiation.cc:407] Connection header received
0925 01:17:57.679584 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:57.679587 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:57.679646 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:57.679724 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:57.680662 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:57.681598 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:57.682842 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:57.683262 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:57.683399 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:57.687112 (+ 3713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:39235: BlockingRecv error: recv got EOF from 127.20.102.129:39235 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 01:17:58 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:58 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 01:17:58.150966 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:58.142556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:48301 (local address 127.20.102.190:33157)
0925 01:17:58.142793 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:58.142799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:58.143041 (+ 242us) server_negotiation.cc:407] Connection header received
0925 01:17:58.143172 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:58.143176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:58.143236 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:58.143314 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:58.144046 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:58.144867 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:58.146051 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:58.146446 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:58.146550 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:58.150784 (+ 4234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:48301: BlockingRecv error: recv got EOF from 127.20.102.130:48301 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 01:17:58.151014 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:58.142414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:48301)
0925 01:17:58.142948 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:58.142966 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:58.143081 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:58.143360 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:58.143364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:58.143753 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:58.143911 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:58.143920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:58.144992 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:58.144995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:58.145914 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:58.145925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:58.146611 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:58.146616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:58.146727 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:58.147301 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:58.147318 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:58.150643 (+ 3325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":286,"threads_started":1}
Sep 25 01:17:58 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:58 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:58.647275 30163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:58.639205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:56667)
0925 01:17:58.639751 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:58.639772 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:58.639898 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:58.640219 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:58.640222 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:58.640622 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:58.640861 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:58.640870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:58.641853 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:58.641856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:58.642777 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:58.642789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:58.643492 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:58.643497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:58.643630 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:58.644217 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:58.644234 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:58.647030 (+ 2796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":291,"threads_started":1}
W20260925 01:17:58.647444 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:58.639346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:56667 (local address 127.20.102.190:33157)
0925 01:17:58.639559 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:58.639563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:58.639888 (+ 325us) server_negotiation.cc:407] Connection header received
0925 01:17:58.640002 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:58.640007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:58.640090 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:58.640174 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:58.641000 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:58.641739 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:58.642920 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:58.643340 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:58.643438 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:58.647218 (+ 3780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:56667: BlockingRecv error: recv got EOF from 127.20.102.131:56667 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 01:17:58 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:58 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:58.697397 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:58.688011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:59043)
0925 01:17:58.688548 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:58.688564 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:58.688749 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:58.689127 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:58.689130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:58.689484 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:58.689667 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:58.689676 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:58.690730 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:58.690733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:58.691648 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:58.691659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:58.692598 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:58.692605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:58.692805 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:58.693475 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:58.693492 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:58.697063 (+ 3571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":268,"threads_started":1}
W20260925 01:17:58.697722 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:58.688077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:59043 (local address 127.20.102.190:33157)
0925 01:17:58.688419 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:58.688425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:58.688647 (+ 222us) server_negotiation.cc:407] Connection header received
0925 01:17:58.688902 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:58.688908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:58.688982 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:58.689061 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:58.689793 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:58.690606 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:58.691788 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:58.692409 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:58.692570 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:58.697289 (+ 4719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:59043: BlockingRecv error: recv got EOF from 127.20.102.129:59043 (error 108)
Metrics: {"server-negotiator.queue_time_us":189}
Sep 25 01:17:59 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:59 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:59.160039 30165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:59.151977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:36153)
0925 01:17:59.152468 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:59.152485 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:59.152605 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:59.152925 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:59.152928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:59.153289 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:59.153464 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:59.153471 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:59.154481 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:59.154484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:59.155430 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:59.155442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:59.156126 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:59.156132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:59.156299 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:59.156985 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:59.157004 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:59.159802 (+ 2798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":269,"threads_started":1}
W20260925 01:17:59.160233 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:59.152094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:36153 (local address 127.20.102.190:33157)
0925 01:17:59.152268 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:59.152272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:59.152561 (+ 289us) server_negotiation.cc:407] Connection header received
0925 01:17:59.152729 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:59.152732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:59.152793 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:59.152876 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:59.153582 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:59.154360 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:59.155562 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:59.155978 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:59.156086 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:59.159989 (+ 3903us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:36153: BlockingRecv error: recv got EOF from 127.20.102.130:36153 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 01:17:59 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:59 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.131@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:59.656519 30166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:59.648137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.131:38433)
0925 01:17:59.648627 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:59.648649 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:59.648860 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:59.649207 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:59.649211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:59.649634 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:59.649815 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:59.649824 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:59.650848 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:59.650854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:59.651813 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:59.651825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:59.652547 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:59.652553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:59.652769 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:59.653437 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:59.653455 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:59.656173 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":249,"threads_started":1}
W20260925 01:17:59.656646 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:59.648208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:38433 (local address 127.20.102.190:33157)
0925 01:17:59.648461 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:59.648466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:59.648798 (+ 332us) server_negotiation.cc:407] Connection header received
0925 01:17:59.648998 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:59.649004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:59.649073 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:59.649170 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:59.649961 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:59.650697 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:59.651956 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:59.652392 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:59.652532 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:59.656399 (+ 3867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.131:38433: BlockingRecv error: recv got EOF from 127.20.102.131:38433 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 01:17:59 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:17:59 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.129@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:17:59.706969 30167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:17:59.698595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.129:37229)
0925 01:17:59.699114 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 01:17:59.699131 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:17:59.699284 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:17:59.699634 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:17:59.699639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:17:59.700150 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:17:59.700320 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:59.700329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:59.701459 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:59.701465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:59.702381 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:17:59.702424 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:17:59.703165 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:17:59.703170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:17:59.703348 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:59.703895 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:17:59.703912 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:17:59.706617 (+ 2705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":276,"threads_started":1}
W20260925 01:17:59.707068 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:17:59.698704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:37229 (local address 127.20.102.190:33157)
0925 01:17:59.698951 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 01:17:59.698957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:17:59.699216 (+ 259us) server_negotiation.cc:407] Connection header received
0925 01:17:59.699406 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:17:59.699411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:17:59.699481 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:17:59.699592 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:17:59.700476 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:59.701316 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:59.702570 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:17:59.702991 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:17:59.703136 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:17:59.706807 (+ 3671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.129:37229: BlockingRecv error: recv got EOF from 127.20.102.129:37229 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 25 01:18:00 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 01:18:00 dist-test-slave-s12s krb5kdc[29121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.20.102.130@KRBTEST.COM for oryx/127.20.102.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 01:18:00.169410 30168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:00.161271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33157 (local address 127.20.102.130:47079)
0925 01:18:00.161838 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:00.161854 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:00.161959 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:00.162270 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:00.162273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:00.162732 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:18:00.162913 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:00.162922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:00.163857 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:00.163861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:00.164887 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:00.164899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:00.165607 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:00.165611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:00.165714 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:00.166283 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:18:00.166299 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:18:00.169167 (+ 2868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.20.102.190:33157: Server oryx/127.20.102.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":326,"threads_started":1}
W20260925 01:18:00.169574 29859 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:18:00.161325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:47079 (local address 127.20.102.190:33157)
0925 01:18:00.161494 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:00.161499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:00.161924 (+ 425us) server_negotiation.cc:407] Connection header received
0925 01:18:00.162070 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:00.162073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:00.162130 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:00.162225 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:00.163036 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:00.163734 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:00.164999 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:00.165456 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:00.165541 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:00.169358 (+ 3817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.20.102.130:47079: BlockingRecv error: recv got EOF from 127.20.102.130:47079 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:18:00.467531 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 29709
I20260925 01:18:00.479419 30170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:00.468466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.20.102.129:33205)
0925 01:18:00.469071 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:00.469078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:00.469105 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:18:00.469162 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:00.469168 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:00.469388 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:00.469534 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:00.470960 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:00.471924 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:00.473137 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:00.473681 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:00.473816 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:00.476258 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:00.476266 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:00.476280 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:00.476314 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:00.478939 (+ 2625us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.102.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 01:18:00.479192 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49788: Request ticket server kudu/127.20.102.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":316,"threads_started":1}
W20260925 01:18:00.479540 30170 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49788: Request ticket server kudu/127.20.102.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 01:18:00.479658 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 29709: Not authorized: Client connection negotiation failed: client connection to 127.20.102.129:33205: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.102.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 01:18:00.479869 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 29709
I20260925 01:18:00.497936 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 29861
I20260925 01:18:00.510461 30171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:00.498997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.20.102.130:36391)
0925 01:18:00.499633 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:00.499640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:00.499668 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:18:00.499730 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:00.499737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:00.499963 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:00.500120 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:00.501450 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:00.502497 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:00.503689 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:00.504574 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:00.504789 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:00.507222 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:00.507231 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:00.507244 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:00.507273 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:00.509925 (+ 2652us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.102.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 01:18:00.510189 (+ 264us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43090: Request ticket server kudu/127.20.102.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":314,"threads_started":1}
W20260925 01:18:00.510565 30171 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43090: Request ticket server kudu/127.20.102.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 01:18:00.510618 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 29861: Not authorized: Client connection negotiation failed: client connection to 127.20.102.130:36391: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.102.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 01:18:00.510799 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 29861
I20260925 01:18:00.528198 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 29992
I20260925 01:18:00.539711 30172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:00.529242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42988 (local address 127.20.102.131:43119)
0925 01:18:00.529901 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:00.529909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:00.529936 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:18:00.529995 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:00.530001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:00.530215 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:00.530370 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:00.531572 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:00.532594 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:00.533768 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:00.534304 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:00.534401 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:00.536870 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:00.536878 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:00.536891 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:00.536929 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:00.539242 (+ 2313us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.102.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 01:18:00.539477 (+ 235us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42988: Request ticket server kudu/127.20.102.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":352,"threads_started":1}
W20260925 01:18:00.539788 30172 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42988: Request ticket server kudu/127.20.102.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 01:18:00.540019 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 29992: Not authorized: Client connection negotiation failed: client connection to 127.20.102.131:43119: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.20.102.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 01:18:00.540238 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 29992
I20260925 01:18:00.557310 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 29640
I20260925 01:18:00.558653 29702 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:00.675717 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 29640
2026-09-25T01:18:00Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21216 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T01:18:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:18:00Z Disabled control of system clock
I20260925 01:18:00.751085 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39853
--webserver_interface=127.20.102.190
--webserver_port=0
--builtin_ntp_servers=127.20.102.148:45125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:39853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 01:18:01.075834 30177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:01.076128 30177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:01.076288 30177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:01.083513 30177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:18:01.083673 30177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:01.083743 30177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:01.083847 30177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:18:01.083928 30177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:18:01.093209 30177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:45125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:39853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:01.094569 30177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:01.096501 30177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:01.108251 30185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:01.108404 30182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:01.108397 30183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:01.109661 30177 server_base.cc:1034] running on GCE node
I20260925 01:18:01.110500 30177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:01.111887 30177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:01.113085 30177 hybrid_clock.cc:648] HybridClock initialized: now 1790299081113045 us; error 35 us; skew 500 ppm
I20260925 01:18:01.113435 30177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:18:01.115688 30177 webserver.cc:460] Webserver started at http://127.20.102.190:40381/ using document root <none> and password file <none>
I20260925 01:18:01.116375 30177 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:01.116492 30177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:01.116894 30177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:18:01.119828 30177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "47e6fd675be544cfa5330c512da63fc8"
format_stamp: "Formatted at 2026-09-25 01:18:01 on dist-test-slave-s12s"
server_key: "f0d55e0d11d7eb9a90abfd3a4e388439"
server_key_iv: "b3973de86886d44e562658fb1db1a478"
server_key_version: "encryptionkey@0"
I20260925 01:18:01.120638 30177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "47e6fd675be544cfa5330c512da63fc8"
format_stamp: "Formatted at 2026-09-25 01:18:01 on dist-test-slave-s12s"
server_key: "f0d55e0d11d7eb9a90abfd3a4e388439"
server_key_iv: "b3973de86886d44e562658fb1db1a478"
server_key_version: "encryptionkey@0"
I20260925 01:18:01.125669 30177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 01:18:01.129103 30191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:01.130414 30177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:18:01.130612 30177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "47e6fd675be544cfa5330c512da63fc8"
format_stamp: "Formatted at 2026-09-25 01:18:01 on dist-test-slave-s12s"
server_key: "f0d55e0d11d7eb9a90abfd3a4e388439"
server_key_iv: "b3973de86886d44e562658fb1db1a478"
server_key_version: "encryptionkey@0"
I20260925 01:18:01.130823 30177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:01.149030 30177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:01.149855 30177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:01.150156 30177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:01.170249 30177 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:39853
I20260925 01:18:01.170271 30242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:39853 every 8 connection(s)
I20260925 01:18:01.171581 30177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:18:01.175307 30243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:18:01.179029 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 30177
I20260925 01:18:01.179221 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:18:01.180079 20890 external_mini_cluster.cc:1428] Setting key daff74273bfdc1b0ba81d7106412ae13
I20260925 01:18:01.190089 30243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8: Bootstrap starting.
I20260925 01:18:01.193295 30243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8: Neither blocks nor log segments found. Creating new log.
I20260925 01:18:01.194343 30243 log.cc:824] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8: Log is configured to *not* fsync() on all Append() calls
I20260925 01:18:01.197248 30243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8: No bootstrap required, opened a new log
I20260925 01:18:01.200299 30246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:01.182175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37262 (local address 127.20.102.190:39853)
0925 01:18:01.189754 (+ 7579us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:01.189765 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:01.189807 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:18:01.191560 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:01.191584 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:01.192040 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:01.192433 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:01.193113 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:01.194475 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:01.195821 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:01.196498 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:01.196724 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:01.198004 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:01.198012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:01.198026 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:18:01.198532 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:01.198607 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:18:01.198630 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:01.198793 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:01.199028 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 01:18:01.199332 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7091,"thread_start_us":6843,"threads_started":1}
I20260925 01:18:01.205628 30243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e6fd675be544cfa5330c512da63fc8" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39853 } }
I20260925 01:18:01.205955 30243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:18:01.206095 30243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e6fd675be544cfa5330c512da63fc8, State: Initialized, Role: FOLLOWER
I20260925 01:18:01.206670 30243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e6fd675be544cfa5330c512da63fc8" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39853 } }
I20260925 01:18:01.206897 30243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:18:01.207062 30243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:18:01.207234 30243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:18:01.209766 30243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e6fd675be544cfa5330c512da63fc8" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39853 } }
I20260925 01:18:01.210245 30243 leader_election.cc:302] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47e6fd675be544cfa5330c512da63fc8; no voters:
I20260925 01:18:01.210821 30243 leader_election.cc:288] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:18:01.210973 30248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:18:01.211694 30248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [term 1 LEADER]: Becoming Leader. State: Replica: 47e6fd675be544cfa5330c512da63fc8, State: Running, Role: LEADER
I20260925 01:18:01.212229 30248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e6fd675be544cfa5330c512da63fc8" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39853 } }
I20260925 01:18:01.212924 30243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:18:01.215013 30249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47e6fd675be544cfa5330c512da63fc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e6fd675be544cfa5330c512da63fc8" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39853 } } }
I20260925 01:18:01.215783 30249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:18:01.215576 30250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47e6fd675be544cfa5330c512da63fc8. Latest consensus state: current_term: 1 leader_uuid: "47e6fd675be544cfa5330c512da63fc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e6fd675be544cfa5330c512da63fc8" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39853 } } }
I20260925 01:18:01.215891 30250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8 [sys.catalog]: This master's current role is: LEADER
I20260925 01:18:01.217702 30255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:18:01.223043 30255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:18:01.232000 30255 catalog_manager.cc:1347] Generated new cluster ID: c492a40238694485adafa3a0f2bc446a
I20260925 01:18:01.232121 30255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:18:01.242383 30255 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:18:01.243605 30255 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:18:01.251325 30255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47e6fd675be544cfa5330c512da63fc8: Generated new TSK 0
I20260925 01:18:01.252085 30255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:18:01.263162 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39853
--builtin_ntp_servers=127.20.102.148:45125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 01:18:01.616240 30267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:01.616626 30267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:01.616856 30267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:01.624934 30267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:01.625028 30267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:01.625129 30267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:18:01.634208 30267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:45125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:01.635818 30267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:01.637537 30267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:01.650049 30275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:01.650121 30272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:01.650151 30273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:01.650995 30267 server_base.cc:1034] running on GCE node
I20260925 01:18:01.651611 30267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:01.652467 30267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:01.653695 30267 hybrid_clock.cc:648] HybridClock initialized: now 1790299081653644 us; error 60 us; skew 500 ppm
I20260925 01:18:01.653996 30267 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:18:01.661538 30267 webserver.cc:460] Webserver started at http://127.20.102.129:36305/ using document root <none> and password file <none>
I20260925 01:18:01.662240 30267 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:01.662377 30267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:01.662703 30267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:18:01.665514 30267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "e0709be755a642ee82d4d82cd63a451e"
format_stamp: "Formatted at 2026-09-25 01:18:01 on dist-test-slave-s12s"
server_key: "bb14c046927d8429feb93530b8bf1da4"
server_key_iv: "4c4c4e52ec9fb8ec72c7a1b84173ec4e"
server_key_version: "encryptionkey@0"
I20260925 01:18:01.666289 30267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "e0709be755a642ee82d4d82cd63a451e"
format_stamp: "Formatted at 2026-09-25 01:18:01 on dist-test-slave-s12s"
server_key: "bb14c046927d8429feb93530b8bf1da4"
server_key_iv: "4c4c4e52ec9fb8ec72c7a1b84173ec4e"
server_key_version: "encryptionkey@0"
I20260925 01:18:01.671185 30267 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:18:01.674414 30281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:01.675729 30267 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:18:01.675889 30267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "e0709be755a642ee82d4d82cd63a451e"
format_stamp: "Formatted at 2026-09-25 01:18:01 on dist-test-slave-s12s"
server_key: "bb14c046927d8429feb93530b8bf1da4"
server_key_iv: "4c4c4e52ec9fb8ec72c7a1b84173ec4e"
server_key_version: "encryptionkey@0"
I20260925 01:18:01.676081 30267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:01.687762 30267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:01.688484 30267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:01.688819 30267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:01.689710 30267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:18:01.691107 30267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:18:01.691190 30267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:01.691327 30267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:18:01.691417 30267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:01.722947 30267 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:45001
I20260925 01:18:01.722991 30393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:45001 every 8 connection(s)
I20260925 01:18:01.724255 30267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:18:01.729369 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 30267
I20260925 01:18:01.729563 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:18:01.730197 20890 external_mini_cluster.cc:1428] Setting key 913eea6cb857ae03d4931f1a9295378e
I20260925 01:18:01.732825 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39853
--builtin_ntp_servers=127.20.102.148:45125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 01:18:01.736950 30396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:01.726813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:60911 (local address 127.20.102.190:39853)
0925 01:18:01.727246 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:01.727251 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:01.728315 (+ 1064us) server_negotiation.cc:407] Connection header received
0925 01:18:01.730238 (+ 1923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:01.730244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:01.730332 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:01.730454 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:01.731626 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:01.732468 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:01.734171 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:01.734615 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:01.734748 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:01.736093 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:01.736099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:01.736102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:18:01.736119 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:01.736162 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:18:01.736174 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:01.736273 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:01.736489 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 01:18:01.736595 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":238,"threads_started":1}
I20260925 01:18:01.737892 30397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:01.727283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39853 (local address 127.20.102.129:60911)
0925 01:18:01.728154 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:01.728197 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:01.729970 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:01.730683 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:01.730691 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:01.730781 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:18:01.731454 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:01.731470 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:01.732636 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:01.732640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:01.733996 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:01.734008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:01.734846 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:01.734855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:01.735086 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:01.735783 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:18:01.735786 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:18:01.735829 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:18:01.735897 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:18:01.736271 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:18:01.736283 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:18:01.736293 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 01:18:01.736480 (+ 187us) client_negotiation.cc:240] Negotiation successful
0925 01:18:01.736868 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":258,"threads_started":1}
I20260925 01:18:01.739385 30394 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39853
I20260925 01:18:01.739683 30394 heartbeater.cc:464] Registering TS with master...
I20260925 01:18:01.740370 30394 heartbeater.cc:511] Master 127.20.102.190:39853 requested a full tablet report, sending...
I20260925 01:18:01.742332 30208 ts_manager.cc:194] Registered new tserver with Master: e0709be755a642ee82d4d82cd63a451e (127.20.102.129:45001)
I20260925 01:18:01.743772 30208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.102.129:60911
W20260925 01:18:02.055442 30398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:02.055720 30398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:02.055859 30398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:02.062927 30398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:02.063033 30398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:02.063252 30398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:18:02.072005 30398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:45125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:02.073285 30398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:02.075096 30398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:02.088440 30403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:02.088483 30404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:02.089675 30398 server_base.cc:1034] running on GCE node
W20260925 01:18:02.090245 30406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:02.090955 30398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:02.092029 30398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:02.093220 30398 hybrid_clock.cc:648] HybridClock initialized: now 1790299082093185 us; error 24 us; skew 500 ppm
I20260925 01:18:02.093490 30398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:18:02.095762 30398 webserver.cc:460] Webserver started at http://127.20.102.130:42249/ using document root <none> and password file <none>
I20260925 01:18:02.096767 30398 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:02.096877 30398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:02.097142 30398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:18:02.099812 30398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "8cb0e77480a94ba48a9e311a542301d5"
format_stamp: "Formatted at 2026-09-25 01:18:02 on dist-test-slave-s12s"
server_key: "71888faa536c51168392a7f562a2027a"
server_key_iv: "d590bd2da1ce30303bf39bc308c3ea02"
server_key_version: "encryptionkey@0"
I20260925 01:18:02.100517 30398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "8cb0e77480a94ba48a9e311a542301d5"
format_stamp: "Formatted at 2026-09-25 01:18:02 on dist-test-slave-s12s"
server_key: "71888faa536c51168392a7f562a2027a"
server_key_iv: "d590bd2da1ce30303bf39bc308c3ea02"
server_key_version: "encryptionkey@0"
I20260925 01:18:02.105666 30398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 01:18:02.108948 30412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:02.110242 30398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:18:02.110419 30398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "8cb0e77480a94ba48a9e311a542301d5"
format_stamp: "Formatted at 2026-09-25 01:18:02 on dist-test-slave-s12s"
server_key: "71888faa536c51168392a7f562a2027a"
server_key_iv: "d590bd2da1ce30303bf39bc308c3ea02"
server_key_version: "encryptionkey@0"
I20260925 01:18:02.110630 30398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:02.130290 30398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:02.131078 30398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:02.131405 30398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:02.132323 30398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:18:02.133757 30398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:18:02.133884 30398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:02.134016 30398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:18:02.134109 30398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:02.168283 30398 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:43625
I20260925 01:18:02.168320 30524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:43625 every 8 connection(s)
I20260925 01:18:02.169584 30398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:18:02.174824 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 30398
I20260925 01:18:02.175004 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:18:02.175588 20890 external_mini_cluster.cc:1428] Setting key 5ba2a58079467b3ca9b88ddf48882850
I20260925 01:18:02.178645 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39853
--builtin_ntp_servers=127.20.102.148:45125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 01:18:02.183030 30396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:02.172298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:49919 (local address 127.20.102.190:39853)
0925 01:18:02.172472 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:02.172477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:02.174170 (+ 1693us) server_negotiation.cc:407] Connection header received
0925 01:18:02.176112 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:02.176118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:02.176208 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:02.176320 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:02.177762 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:02.178502 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:02.180288 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:02.180827 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:02.180954 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:02.182243 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:02.182249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:02.182254 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:18:02.182281 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:02.182310 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:18:02.182321 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:02.182422 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:02.182645 (+ 223us) server_negotiation.cc:299] Negotiation successful
0925 01:18:02.182736 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:18:02.184039 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:02.172935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39853 (local address 127.20.102.130:49919)
0925 01:18:02.173981 (+ 1046us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:02.174026 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:02.175846 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:02.176574 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:02.176585 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:02.176718 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:18:02.177523 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:02.177559 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:02.178659 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:02.178664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:02.180136 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:02.180148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:02.181046 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:02.181053 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:02.181264 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:02.181954 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:18:02.181957 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:18:02.181991 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:18:02.182066 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:18:02.182435 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:18:02.182447 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:18:02.182460 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 01:18:02.182633 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 01:18:02.182996 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":307,"threads_started":1}
I20260925 01:18:02.185434 30525 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39853
I20260925 01:18:02.185737 30525 heartbeater.cc:464] Registering TS with master...
I20260925 01:18:02.186444 30525 heartbeater.cc:511] Master 127.20.102.190:39853 requested a full tablet report, sending...
I20260925 01:18:02.188091 30208 ts_manager.cc:194] Registered new tserver with Master: 8cb0e77480a94ba48a9e311a542301d5 (127.20.102.130:43625)
I20260925 01:18:02.189239 30208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.102.130:49919
W20260925 01:18:02.503032 30528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:02.503355 30528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:02.503513 30528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:02.510511 30528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:02.510635 30528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:02.510798 30528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:18:02.520610 30528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:45125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:02.522065 30528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:02.523869 30528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:02.538767 30534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:02.538851 30533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:02.538795 30536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:02.539693 30528 server_base.cc:1034] running on GCE node
I20260925 01:18:02.540259 30528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:02.541271 30528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:02.542474 30528 hybrid_clock.cc:648] HybridClock initialized: now 1790299082542423 us; error 47 us; skew 500 ppm
I20260925 01:18:02.542811 30528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:18:02.545094 30528 webserver.cc:460] Webserver started at http://127.20.102.131:38931/ using document root <none> and password file <none>
I20260925 01:18:02.545778 30528 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:02.545894 30528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:02.546231 30528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:18:02.549079 30528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "f72e3a22d4dd45e5a2846a55207d0aba"
format_stamp: "Formatted at 2026-09-25 01:18:02 on dist-test-slave-s12s"
server_key: "ba6ffe9bf3ae40faed4eab49a1ec84da"
server_key_iv: "5a8995f5094ca03772464bde3de76f3d"
server_key_version: "encryptionkey@0"
I20260925 01:18:02.549894 30528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "f72e3a22d4dd45e5a2846a55207d0aba"
format_stamp: "Formatted at 2026-09-25 01:18:02 on dist-test-slave-s12s"
server_key: "ba6ffe9bf3ae40faed4eab49a1ec84da"
server_key_iv: "5a8995f5094ca03772464bde3de76f3d"
server_key_version: "encryptionkey@0"
I20260925 01:18:02.555018 30528 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:18:02.558447 30542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:02.559762 30528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:18:02.559942 30528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "f72e3a22d4dd45e5a2846a55207d0aba"
format_stamp: "Formatted at 2026-09-25 01:18:02 on dist-test-slave-s12s"
server_key: "ba6ffe9bf3ae40faed4eab49a1ec84da"
server_key_iv: "5a8995f5094ca03772464bde3de76f3d"
server_key_version: "encryptionkey@0"
I20260925 01:18:02.560164 30528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:02.578388 30528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:02.579280 30528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:02.579608 30528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:02.580585 30528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:18:02.582053 30528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:18:02.582173 30528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:02.582309 30528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:18:02.582397 30528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:02.614032 30528 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:39649
I20260925 01:18:02.614118 30654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:39649 every 8 connection(s)
I20260925 01:18:02.615458 30528 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:18:02.624079 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 30528
I20260925 01:18:02.624260 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:18:02.624857 20890 external_mini_cluster.cc:1428] Setting key 9045d4b1d9846ad0c76481638bc6aef0
I20260925 01:18:02.628724 30396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:02.618049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:52517 (local address 127.20.102.190:39853)
0925 01:18:02.618226 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:02.618231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:02.619681 (+ 1450us) server_negotiation.cc:407] Connection header received
0925 01:18:02.621448 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:02.621452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:02.621522 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:02.621611 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:02.622761 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:02.623515 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:02.625816 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:02.626354 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:02.626448 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:02.627916 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:02.627922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:02.627928 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:18:02.627952 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:02.628023 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:18:02.628041 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:02.628145 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:02.628350 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 01:18:02.628426 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:18:02.629694 30657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:02.618566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39853 (local address 127.20.102.131:52517)
0925 01:18:02.619518 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:02.619562 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:02.621205 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:02.621833 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:02.621841 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:02.621925 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:18:02.622607 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:02.622627 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:02.623740 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:02.623747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:02.625679 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:02.625696 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:02.626613 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:02.626624 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:02.626829 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:02.627545 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:18:02.627550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:18:02.627603 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:18:02.627700 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:18:02.628158 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:18:02.628167 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:18:02.628174 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 01:18:02.628320 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 01:18:02.628609 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":308,"threads_started":1}
I20260925 01:18:02.631062 30655 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39853
I20260925 01:18:02.631352 30655 heartbeater.cc:464] Registering TS with master...
I20260925 01:18:02.632107 30655 heartbeater.cc:511] Master 127.20.102.190:39853 requested a full tablet report, sending...
I20260925 01:18:02.633500 30208 ts_manager.cc:194] Registered new tserver with Master: f72e3a22d4dd45e5a2846a55207d0aba (127.20.102.131:39649)
I20260925 01:18:02.634343 30208 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.102.131:52517
I20260925 01:18:02.640165 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 01:18:02.646668 30396 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:18:02.644880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.20.102.190:39853)
0925 01:18:02.645041 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:02.645045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:02.645550 (+ 505us) server_negotiation.cc:407] Connection header received
0925 01:18:02.645757 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:02.645767 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:02.645839 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:02.645942 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:02.646397 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35216: BlockingRecv error: recv got EOF from 127.0.0.1:35216 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:18:02.647485 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 30267
I20260925 01:18:02.655041 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:02.648474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.20.102.129:45001)
0925 01:18:02.649057 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:02.649063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:02.649088 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:18:02.649141 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:02.649146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:02.649339 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:02.649469 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:02.649831 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:02.650822 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:02.652042 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:02.652581 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:02.652739 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:02.653671 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:02.653679 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:02.653691 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:18:02.654229 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:02.654273 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:18:02.654292 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:02.654387 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:02.654500 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 01:18:02.654669 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":317,"threads_started":1}
I20260925 01:18:02.656167 30389 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:02.737874 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 30267
I20260925 01:18:02.754779 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 30398
I20260925 01:18:02.762400 30666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:02.755705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.20.102.130:43625)
0925 01:18:02.756308 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:02.756315 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:02.756343 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:18:02.756411 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:02.756418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:02.756639 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:02.756826 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:02.757176 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:02.758162 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:02.759324 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:02.759877 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:02.759971 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:02.761066 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:02.761074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:02.761087 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:18:02.761680 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:02.761726 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:18:02.761745 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:02.761835 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:02.761992 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 01:18:02.762111 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":317,"threads_started":1}
I20260925 01:18:02.763641 30520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:02.845594 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 30398
I20260925 01:18:02.862491 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 30528
I20260925 01:18:02.871083 30668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:02.863421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.20.102.131:39649)
0925 01:18:02.864138 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:02.864145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:02.864175 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:18:02.864242 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:02.864249 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:02.864463 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:02.864683 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:02.865215 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:02.866480 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:02.867969 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:02.868533 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:02.868630 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:02.869672 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:02.869680 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:02.869693 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:18:02.870285 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:02.870332 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:18:02.870350 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:02.870436 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:02.870567 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 01:18:02.870723 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":388,"threads_started":1}
I20260925 01:18:02.872493 30650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:02.961930 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 30528
I20260925 01:18:02.979139 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 30177
I20260925 01:18:02.980430 30238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:03.084857 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 30177
2026-09-25T01:18:03Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2408 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T01:18:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:18:03Z Disabled control of system clock
I20260925 01:18:03.161113 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39301
--webserver_interface=127.20.102.190
--webserver_port=0
--builtin_ntp_servers=127.20.102.148:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:39301
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 01:18:03.536091 30674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:03.536458 30674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:03.536633 30674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:03.544018 30674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:18:03.544166 30674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:03.544268 30674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:03.544385 30674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:18:03.544489 30674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:18:03.553941 30674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:39301
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:03.555279 30674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:03.557183 30674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:03.568655 30680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:03.568791 30682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:03.568713 30679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:03.570401 30674 server_base.cc:1034] running on GCE node
I20260925 01:18:03.571194 30674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:03.572563 30674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:03.573787 30674 hybrid_clock.cc:648] HybridClock initialized: now 1790299083573724 us; error 51 us; skew 500 ppm
I20260925 01:18:03.574133 30674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:18:03.576356 30674 webserver.cc:460] Webserver started at http://127.20.102.190:35999/ using document root <none> and password file <none>
I20260925 01:18:03.577082 30674 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:03.577188 30674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:03.577510 30674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:18:03.580330 30674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "69d0361369e443fbb33972083d5859cc"
format_stamp: "Formatted at 2026-09-25 01:18:03 on dist-test-slave-s12s"
server_key: "5764e8c878a7ee55582ed6397bfae50c"
server_key_iv: "165ef655cb7ec7c230d6416cc8c57deb"
server_key_version: "encryptionkey@0"
I20260925 01:18:03.581126 30674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "69d0361369e443fbb33972083d5859cc"
format_stamp: "Formatted at 2026-09-25 01:18:03 on dist-test-slave-s12s"
server_key: "5764e8c878a7ee55582ed6397bfae50c"
server_key_iv: "165ef655cb7ec7c230d6416cc8c57deb"
server_key_version: "encryptionkey@0"
I20260925 01:18:03.586009 30674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 01:18:03.589696 30688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:03.591099 30674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:18:03.591289 30674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "69d0361369e443fbb33972083d5859cc"
format_stamp: "Formatted at 2026-09-25 01:18:03 on dist-test-slave-s12s"
server_key: "5764e8c878a7ee55582ed6397bfae50c"
server_key_iv: "165ef655cb7ec7c230d6416cc8c57deb"
server_key_version: "encryptionkey@0"
I20260925 01:18:03.591498 30674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:03.597223 30674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:03.597952 30674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:03.598235 30674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:03.617183 30674 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:39301
I20260925 01:18:03.617228 30739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:39301 every 8 connection(s)
I20260925 01:18:03.618495 30674 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:18:03.620983 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 30674
I20260925 01:18:03.621205 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:18:03.621994 20890 external_mini_cluster.cc:1428] Setting key 7d4ec2e2528dc47f7204fc1351d0cf26
I20260925 01:18:03.622910 30740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:18:03.631878 30743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:03.624292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.20.102.190:39301)
0925 01:18:03.625404 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:03.625420 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:03.625474 (+ 54us) server_negotiation.cc:407] Connection header received
0925 01:18:03.627812 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:03.627840 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:03.628372 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:03.628845 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:03.629867 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:03.629875 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:03.629897 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:18:03.629913 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:03.629991 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:18:03.630010 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:03.630236 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:03.630368 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 01:18:03.630671 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":393,"threads_started":1}
I20260925 01:18:03.637832 30740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc: Bootstrap starting.
I20260925 01:18:03.640669 30740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc: Neither blocks nor log segments found. Creating new log.
I20260925 01:18:03.641671 30740 log.cc:824] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc: Log is configured to *not* fsync() on all Append() calls
I20260925 01:18:03.644459 30740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc: No bootstrap required, opened a new log
I20260925 01:18:03.651733 30740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d0361369e443fbb33972083d5859cc" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39301 } }
I20260925 01:18:03.652064 30740 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:18:03.652136 30740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d0361369e443fbb33972083d5859cc, State: Initialized, Role: FOLLOWER
I20260925 01:18:03.652611 30740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d0361369e443fbb33972083d5859cc" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39301 } }
I20260925 01:18:03.652971 30740 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:18:03.653139 30740 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:18:03.653311 30740 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:18:03.655433 30740 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d0361369e443fbb33972083d5859cc" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39301 } }
I20260925 01:18:03.655905 30740 leader_election.cc:302] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69d0361369e443fbb33972083d5859cc; no voters:
I20260925 01:18:03.656605 30740 leader_election.cc:288] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:18:03.656874 30745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:18:03.657833 30745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [term 1 LEADER]: Becoming Leader. State: Replica: 69d0361369e443fbb33972083d5859cc, State: Running, Role: LEADER
I20260925 01:18:03.658450 30745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d0361369e443fbb33972083d5859cc" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39301 } }
I20260925 01:18:03.658845 30740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:18:03.661897 30746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69d0361369e443fbb33972083d5859cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d0361369e443fbb33972083d5859cc" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39301 } } }
I20260925 01:18:03.662412 30746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [sys.catalog]: This master's current role is: LEADER
I20260925 01:18:03.661906 30747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69d0361369e443fbb33972083d5859cc. Latest consensus state: current_term: 1 leader_uuid: "69d0361369e443fbb33972083d5859cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69d0361369e443fbb33972083d5859cc" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39301 } } }
I20260925 01:18:03.662848 30747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc [sys.catalog]: This master's current role is: LEADER
I20260925 01:18:03.663919 30753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:18:03.669538 30753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:18:03.678471 30753 catalog_manager.cc:1347] Generated new cluster ID: d7a469d6c3984db89ce6c559f7734756
I20260925 01:18:03.678603 30753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:18:03.696355 30753 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:18:03.696933 30753 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:18:03.703616 30753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69d0361369e443fbb33972083d5859cc: Generated new TSK 0
I20260925 01:18:03.704417 30753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:18:03.722949 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39301
--builtin_ntp_servers=127.20.102.148:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 01:18:04.071704 30764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:04.071954 30764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:04.072080 30764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:04.079355 30764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:04.079449 30764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:04.079620 30764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:18:04.088846 30764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:04.090152 30764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:04.091813 30764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:04.105831 30772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:04.105938 30770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:04.105861 30769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:04.114238 30764 server_base.cc:1034] running on GCE node
I20260925 01:18:04.114864 30764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:04.115852 30764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:04.117093 30764 hybrid_clock.cc:648] HybridClock initialized: now 1790299084117057 us; error 54 us; skew 500 ppm
I20260925 01:18:04.117430 30764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:18:04.120035 30764 webserver.cc:460] Webserver started at http://127.20.102.129:42697/ using document root <none> and password file <none>
I20260925 01:18:04.120867 30764 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:04.120996 30764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:04.121341 30764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:18:04.124210 30764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "4bf033644cd54fa1b0d76eed574b2d95"
format_stamp: "Formatted at 2026-09-25 01:18:04 on dist-test-slave-s12s"
server_key: "2cc7d57955310b289f606e389246a013"
server_key_iv: "5a4143c84396a354db38417c10eb063f"
server_key_version: "encryptionkey@0"
I20260925 01:18:04.125185 30764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "4bf033644cd54fa1b0d76eed574b2d95"
format_stamp: "Formatted at 2026-09-25 01:18:04 on dist-test-slave-s12s"
server_key: "2cc7d57955310b289f606e389246a013"
server_key_iv: "5a4143c84396a354db38417c10eb063f"
server_key_version: "encryptionkey@0"
I20260925 01:18:04.130294 30764 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:18:04.133978 30778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:04.135346 30764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:18:04.135527 30764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "4bf033644cd54fa1b0d76eed574b2d95"
format_stamp: "Formatted at 2026-09-25 01:18:04 on dist-test-slave-s12s"
server_key: "2cc7d57955310b289f606e389246a013"
server_key_iv: "5a4143c84396a354db38417c10eb063f"
server_key_version: "encryptionkey@0"
I20260925 01:18:04.135769 30764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:04.141556 30764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:04.142313 30764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:04.142585 30764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:04.143497 30764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:18:04.144935 30764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:18:04.145057 30764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:04.145187 30764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:18:04.145274 30764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:04.178982 30764 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:40927
I20260925 01:18:04.179073 30890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:40927 every 8 connection(s)
I20260925 01:18:04.180392 30764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:18:04.188731 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:04.183160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:60811 (local address 127.20.102.190:39301)
0925 01:18:04.183613 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:04.183620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:04.184958 (+ 1338us) server_negotiation.cc:407] Connection header received
0925 01:18:04.186626 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:04.186630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:04.186680 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:04.186777 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:04.188044 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:04.188048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:04.188051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:18:04.188067 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:04.188110 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:18:04.188121 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:04.188220 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:04.188394 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 01:18:04.188444 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":199,"threads_started":1}
I20260925 01:18:04.189602 30894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:04.183827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39301 (local address 127.20.102.129:60811)
0925 01:18:04.184808 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:04.184849 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:04.186385 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:04.186983 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:04.186990 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:04.187077 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:18:04.187767 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:18:04.187769 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:18:04.187805 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:18:04.187891 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:18:04.188212 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:18:04.188219 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:18:04.188225 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 01:18:04.188437 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 01:18:04.188625 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":265,"threads_started":1}
I20260925 01:18:04.190536 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 30764
I20260925 01:18:04.190752 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:18:04.191100 30891 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39301
I20260925 01:18:04.191442 30891 heartbeater.cc:464] Registering TS with master...
I20260925 01:18:04.191582 20890 external_mini_cluster.cc:1428] Setting key 06edff537f1b2102b54a4412b86c8a39
I20260925 01:18:04.192163 30891 heartbeater.cc:511] Master 127.20.102.190:39301 requested a full tablet report, sending...
I20260925 01:18:04.194489 30704 ts_manager.cc:194] Registered new tserver with Master: 4bf033644cd54fa1b0d76eed574b2d95 (127.20.102.129:40927)
I20260925 01:18:04.194689 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39301
--builtin_ntp_servers=127.20.102.148:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 01:18:04.537171 30895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:04.537428 30895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:04.537619 30895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:04.544828 30895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:04.544942 30895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:04.545140 30895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:18:04.554090 30895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:04.555382 30895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:04.557147 30895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:04.570253 30901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:04.570297 30903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:04.570334 30900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:04.571338 30895 server_base.cc:1034] running on GCE node
I20260925 01:18:04.571949 30895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:04.572990 30895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:04.574219 30895 hybrid_clock.cc:648] HybridClock initialized: now 1790299084574183 us; error 50 us; skew 500 ppm
I20260925 01:18:04.574546 30895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:18:04.576849 30895 webserver.cc:460] Webserver started at http://127.20.102.130:46687/ using document root <none> and password file <none>
I20260925 01:18:04.577610 30895 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:04.577741 30895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:04.578063 30895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:18:04.580994 30895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "27b1289355094f739b031cd6b52e2b62"
format_stamp: "Formatted at 2026-09-25 01:18:04 on dist-test-slave-s12s"
server_key: "fa5eccb094e832df4957fd3cad6f5071"
server_key_iv: "89ee9faa46b8481a7a90fe1205a111b3"
server_key_version: "encryptionkey@0"
I20260925 01:18:04.581768 30895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "27b1289355094f739b031cd6b52e2b62"
format_stamp: "Formatted at 2026-09-25 01:18:04 on dist-test-slave-s12s"
server_key: "fa5eccb094e832df4957fd3cad6f5071"
server_key_iv: "89ee9faa46b8481a7a90fe1205a111b3"
server_key_version: "encryptionkey@0"
I20260925 01:18:04.586763 30895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 01:18:04.590016 30909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:04.591228 30895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 01:18:04.591391 30895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "27b1289355094f739b031cd6b52e2b62"
format_stamp: "Formatted at 2026-09-25 01:18:04 on dist-test-slave-s12s"
server_key: "fa5eccb094e832df4957fd3cad6f5071"
server_key_iv: "89ee9faa46b8481a7a90fe1205a111b3"
server_key_version: "encryptionkey@0"
I20260925 01:18:04.591616 30895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:04.597182 30895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:04.597963 30895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:04.598187 30895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:04.599020 30895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:18:04.600335 30895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:18:04.600419 30895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:04.600508 30895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:18:04.600564 30895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:04.635058 30895 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:39367
I20260925 01:18:04.635104 31021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:39367 every 8 connection(s)
I20260925 01:18:04.636382 30895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:18:04.637243 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 30895
I20260925 01:18:04.637436 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:18:04.638023 20890 external_mini_cluster.cc:1428] Setting key d074e69abec218f5637dd71687457a5b
I20260925 01:18:04.641438 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39301
--builtin_ntp_servers=127.20.102.148:34809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 01:18:04.645287 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:04.639434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:55731 (local address 127.20.102.190:39301)
0925 01:18:04.639648 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:04.639654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:04.641301 (+ 1647us) server_negotiation.cc:407] Connection header received
0925 01:18:04.642962 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:04.642971 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:04.643050 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:04.643157 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:04.644460 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:04.644468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:04.644475 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:18:04.644505 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:04.644546 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:18:04.644563 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:04.644713 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:04.644920 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 01:18:04.644984 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 01:18:04.646071 31024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:04.640010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39301 (local address 127.20.102.130:55731)
0925 01:18:04.641144 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:04.641188 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:04.642711 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:04.643334 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:04.643341 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:04.643427 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:18:04.644165 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:18:04.644168 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:18:04.644204 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:18:04.644285 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:18:04.644661 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:18:04.644668 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:18:04.644674 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 01:18:04.644953 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 01:18:04.645135 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":379,"threads_started":1}
I20260925 01:18:04.647442 31022 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39301
I20260925 01:18:04.647790 31022 heartbeater.cc:464] Registering TS with master...
I20260925 01:18:04.648334 31022 heartbeater.cc:511] Master 127.20.102.190:39301 requested a full tablet report, sending...
I20260925 01:18:04.649821 30704 ts_manager.cc:194] Registered new tserver with Master: 27b1289355094f739b031cd6b52e2b62 (127.20.102.130:39367)
W20260925 01:18:04.966867 31025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:04.967126 31025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:04.967213 31025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:04.974526 31025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:04.974622 31025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:04.974730 31025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:18:04.983796 31025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:04.985071 31025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:04.986763 31025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:04.999677 31031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:04.999809 31033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:04.999979 31030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:05.000777 31025 server_base.cc:1034] running on GCE node
I20260925 01:18:05.001446 31025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:05.002476 31025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:05.003726 31025 hybrid_clock.cc:648] HybridClock initialized: now 1790299085003694 us; error 33 us; skew 500 ppm
I20260925 01:18:05.004076 31025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:18:05.011689 31025 webserver.cc:460] Webserver started at http://127.20.102.131:33121/ using document root <none> and password file <none>
I20260925 01:18:05.012387 31025 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:05.012509 31025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:05.012871 31025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:18:05.015611 31025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "4f4e380ff0c3445da23753d771d4ded2"
format_stamp: "Formatted at 2026-09-25 01:18:05 on dist-test-slave-s12s"
server_key: "8ac796e7ebc91c3a03a1b9923b09202a"
server_key_iv: "61186587d2f6d5d2d2ae785cb7d96d49"
server_key_version: "encryptionkey@0"
I20260925 01:18:05.016371 31025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "4f4e380ff0c3445da23753d771d4ded2"
format_stamp: "Formatted at 2026-09-25 01:18:05 on dist-test-slave-s12s"
server_key: "8ac796e7ebc91c3a03a1b9923b09202a"
server_key_iv: "61186587d2f6d5d2d2ae785cb7d96d49"
server_key_version: "encryptionkey@0"
I20260925 01:18:05.021246 31025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:18:05.024349 31039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:05.025635 31025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 01:18:05.025799 31025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "4f4e380ff0c3445da23753d771d4ded2"
format_stamp: "Formatted at 2026-09-25 01:18:05 on dist-test-slave-s12s"
server_key: "8ac796e7ebc91c3a03a1b9923b09202a"
server_key_iv: "61186587d2f6d5d2d2ae785cb7d96d49"
server_key_version: "encryptionkey@0"
I20260925 01:18:05.025998 31025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:05.031492 31025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:05.032219 31025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:05.032474 31025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:05.033383 31025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:18:05.034678 31025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:18:05.034795 31025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:05.034931 31025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:18:05.035022 31025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:05.067683 31025 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:36761
I20260925 01:18:05.067739 31151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:36761 every 8 connection(s)
I20260925 01:18:05.069079 31025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:18:05.073627 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 31025
I20260925 01:18:05.073820 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:18:05.074234 20890 external_mini_cluster.cc:1428] Setting key a0edbccdc1e33610298b93b811230a00
I20260925 01:18:05.077364 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:05.071660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:44271 (local address 127.20.102.190:39301)
0925 01:18:05.071833 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:05.071837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:05.073360 (+ 1523us) server_negotiation.cc:407] Connection header received
0925 01:18:05.075089 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:05.075094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:05.075158 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:05.075303 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:05.076591 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:05.076597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:05.076602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:18:05.076619 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:05.076659 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:18:05.076672 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:05.076824 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:05.077024 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 01:18:05.077092 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:18:05.078198 31154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:05.072267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39301 (local address 127.20.102.131:44271)
0925 01:18:05.073201 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:05.073237 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:05.074850 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:05.075525 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:05.075533 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:05.075622 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:18:05.076296 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:18:05.076298 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:18:05.076335 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:18:05.076415 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:18:05.076793 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:18:05.076806 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:18:05.076814 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 01:18:05.077099 (+ 285us) client_negotiation.cc:240] Negotiation successful
0925 01:18:05.077334 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":253,"threads_started":1}
I20260925 01:18:05.079427 31152 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39301
I20260925 01:18:05.079710 31152 heartbeater.cc:464] Registering TS with master...
I20260925 01:18:05.080248 31152 heartbeater.cc:511] Master 127.20.102.190:39301 requested a full tablet report, sending...
I20260925 01:18:05.081583 30704 ts_manager.cc:194] Registered new tserver with Master: 4f4e380ff0c3445da23753d771d4ded2 (127.20.102.131:36761)
I20260925 01:18:05.089327 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 01:18:05.095372 30893 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:18:05.093884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49880 (local address 127.20.102.190:39301)
0925 01:18:05.094079 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:05.094087 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:05.094528 (+ 441us) server_negotiation.cc:407] Connection header received
0925 01:18:05.094735 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:05.094743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:05.094814 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:05.094882 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:05.095125 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49880: BlockingRecv error: recv got EOF from 127.0.0.1:49880 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:18:05.096220 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 30764
I20260925 01:18:05.099584 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:05.097269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37792 (local address 127.20.102.129:40927)
0925 01:18:05.097737 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:05.097743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:05.097766 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:18:05.097816 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:05.097822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:05.097983 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:05.098112 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:05.099006 (+ 894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:05.099011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:05.099028 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:18:05.099064 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:05.099100 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:18:05.099113 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:05.099213 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:05.099295 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 01:18:05.099365 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":212,"threads_started":1}
I20260925 01:18:05.100831 30886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:05.197139 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 30764
I20260925 01:18:05.214560 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 30895
I20260925 01:18:05.218146 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:05.215530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39358 (local address 127.20.102.130:39367)
0925 01:18:05.216102 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:05.216109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:05.216137 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:18:05.216201 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:05.216207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:05.216459 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:05.216592 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:05.217519 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:05.217527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:05.217539 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:18:05.217558 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:05.217602 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:18:05.217616 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:05.217721 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:05.217808 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 01:18:05.217893 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":283,"threads_started":1}
I20260925 01:18:05.219355 31017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:05.299654 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 30895
I20260925 01:18:05.316612 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 31025
I20260925 01:18:05.320463 31165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:05.317722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.20.102.131:36761)
0925 01:18:05.318378 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:05.318385 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:05.318416 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:18:05.318477 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:05.318483 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:05.318720 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:05.318849 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:05.319761 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:05.319767 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:05.319778 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:18:05.319790 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:05.319826 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:18:05.319840 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:05.319970 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:05.320021 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 01:18:05.320129 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":349,"threads_started":1}
I20260925 01:18:05.321812 31147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:05.403963 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 31025
I20260925 01:18:05.420450 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 30674
I20260925 01:18:05.421940 30735 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:05.530700 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 30674
2026-09-25T01:18:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2440 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:18:05 dist-test-slave-s12s krb5kdc[31170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:18:05 dist-test-slave-s12s krb5kdc[31170](info): set up 2 sockets
Sep 25 01:18:05 dist-test-slave-s12s krb5kdc[31170](info): commencing operation
krb5kdc: starting...
W20260925 01:18:06.653752 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:18:06 dist-test-slave-s12s krb5kdc[31170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:18:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:18:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:18:06.868027 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:42697
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:42697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:18:07.196626 31183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:07.196913 31183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:07.197088 31183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:07.204147 31183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:18:07.204239 31183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:07.204339 31183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:07.204433 31183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:18:07.204516 31183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:18:07.214073 31183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:42697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:42697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:07.215348 31183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:07.216666 31183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:07.228143 31188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:07.228042 31191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:07.228253 31189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:07.229180 31183 server_base.cc:1034] running on GCE node
I20260925 01:18:07.229961 31183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:07.231276 31183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:07.232473 31183 hybrid_clock.cc:648] HybridClock initialized: now 1790299087232429 us; error 32 us; skew 500 ppm
I20260925 01:18:07.232882 31183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:18:07 dist-test-slave-s12s krb5kdc[31170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299087, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:18:07.240404 31183 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:18:07.241888 31183 webserver.cc:460] Webserver started at http://127.20.102.190:36611/ using document root <none> and password file <none>
I20260925 01:18:07.242558 31183 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:07.242703 31183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:07.243013 31183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:18:07.245968 31183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "9f51024c15eb4999b2f2a1070b16b7b1"
format_stamp: "Formatted at 2026-09-25 01:18:07 on dist-test-slave-s12s"
server_key: "187e1aa480ca58ec4adc6cba9f36329c"
server_key_iv: "9975efa3397982463be21339e6b6bed0"
server_key_version: "encryptionkey@0"
I20260925 01:18:07.246762 31183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "9f51024c15eb4999b2f2a1070b16b7b1"
format_stamp: "Formatted at 2026-09-25 01:18:07 on dist-test-slave-s12s"
server_key: "187e1aa480ca58ec4adc6cba9f36329c"
server_key_iv: "9975efa3397982463be21339e6b6bed0"
server_key_version: "encryptionkey@0"
I20260925 01:18:07.251647 31183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:18:07.254942 31198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:07.256196 31183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 01:18:07.256385 31183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "9f51024c15eb4999b2f2a1070b16b7b1"
format_stamp: "Formatted at 2026-09-25 01:18:07 on dist-test-slave-s12s"
server_key: "187e1aa480ca58ec4adc6cba9f36329c"
server_key_iv: "9975efa3397982463be21339e6b6bed0"
server_key_version: "encryptionkey@0"
I20260925 01:18:07.256582 31183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:07.274667 31183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:07.278718 31183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:07.279004 31183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:07.297283 31183 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:42697
I20260925 01:18:07.297286 31249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:42697 every 8 connection(s)
I20260925 01:18:07.298776 31183 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:18:07.302321 31250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:18:07.307667 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 31183
I20260925 01:18:07.307900 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:18:07.308725 20890 external_mini_cluster.cc:1428] Setting key 3254308eaae072c660f64690b51c18b6
I20260925 01:18:07.313681 31250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1: Bootstrap starting.
I20260925 01:18:07.316603 31250 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1: Neither blocks nor log segments found. Creating new log.
I20260925 01:18:07.317584 31250 log.cc:824] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1: Log is configured to *not* fsync() on all Append() calls
I20260925 01:18:07.320782 31250 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1: No bootstrap required, opened a new log
Sep 25 01:18:07 dist-test-slave-s12s krb5kdc[31170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:18:07.331172 31250 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f51024c15eb4999b2f2a1070b16b7b1" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42697 } }
I20260925 01:18:07.331593 31250 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:18:07.331698 31250 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f51024c15eb4999b2f2a1070b16b7b1, State: Initialized, Role: FOLLOWER
I20260925 01:18:07.332337 31250 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f51024c15eb4999b2f2a1070b16b7b1" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42697 } }
I20260925 01:18:07.332599 31250 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:18:07.332778 31250 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:18:07.332818 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:07.310841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46010 (local address 127.20.102.190:42697)
0925 01:18:07.311641 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:07.311655 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:07.311698 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:18:07.313403 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:07.313425 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:07.313839 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:07.314236 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:07.315766 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:07.317162 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:07.318829 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:07.319817 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:07.320007 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:07.324360 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:07.324370 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:07.324389 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:07.324440 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:07.328494 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:07.329238 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:07.329247 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:07.329255 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:07.329480 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:07.329830 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:07.329835 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:07.329839 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:07.330505 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:07.330817 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:07.331077 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 01:18:07.331494 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":377,"threads_started":1}
I20260925 01:18:07.332984 31250 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:18:07.335744 31250 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f51024c15eb4999b2f2a1070b16b7b1" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42697 } }
I20260925 01:18:07.336150 31250 leader_election.cc:302] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f51024c15eb4999b2f2a1070b16b7b1; no voters:
I20260925 01:18:07.336668 31250 leader_election.cc:288] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:18:07.337015 31255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:18:07.337731 31255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [term 1 LEADER]: Becoming Leader. State: Replica: 9f51024c15eb4999b2f2a1070b16b7b1, State: Running, Role: LEADER
I20260925 01:18:07.338294 31255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f51024c15eb4999b2f2a1070b16b7b1" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42697 } }
I20260925 01:18:07.338758 31250 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:18:07.341058 31257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f51024c15eb4999b2f2a1070b16b7b1. Latest consensus state: current_term: 1 leader_uuid: "9f51024c15eb4999b2f2a1070b16b7b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f51024c15eb4999b2f2a1070b16b7b1" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42697 } } }
I20260925 01:18:07.341498 31257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [sys.catalog]: This master's current role is: LEADER
I20260925 01:18:07.342099 31256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f51024c15eb4999b2f2a1070b16b7b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f51024c15eb4999b2f2a1070b16b7b1" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42697 } } }
I20260925 01:18:07.342461 31256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1 [sys.catalog]: This master's current role is: LEADER
I20260925 01:18:07.349148 31260 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:18:07.353787 31260 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:18:07.361686 31260 catalog_manager.cc:1347] Generated new cluster ID: 248d95222ee94158b3f241c9c1207efd
I20260925 01:18:07.361780 31260 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:18:07.372243 31260 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:18:07.373539 31260 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:18:07.385668 31260 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1: Generated new TSK 0
I20260925 01:18:07.386385 31260 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:18:07.442526 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:42697
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:18:07.775024 31276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:07.775326 31276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:07.775424 31276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:07.782840 31276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:07.782944 31276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:07.783059 31276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:18:07.792666 31276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:07.794011 31276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:07.795285 31276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:07.809154 31281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:07.809190 31284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:07.809231 31282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:07.810467 31276 server_base.cc:1034] running on GCE node
I20260925 01:18:07.810999 31276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:07.811853 31276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:07.813046 31276 hybrid_clock.cc:648] HybridClock initialized: now 1790299087813000 us; error 35 us; skew 500 ppm
I20260925 01:18:07.813316 31276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:18:07 dist-test-slave-s12s krb5kdc[31170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299087, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:18:07.816524 31276 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:18:07.817953 31276 webserver.cc:460] Webserver started at http://127.20.102.129:43147/ using document root <none> and password file <none>
I20260925 01:18:07.818632 31276 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:07.818740 31276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:07.819104 31276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:18:07.822024 31276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "4ee50451f0b34c66b99a0dfe2abb0e46"
format_stamp: "Formatted at 2026-09-25 01:18:07 on dist-test-slave-s12s"
server_key: "6421617b5e83ae79c09fb07b06412a3c"
server_key_iv: "2ca5dfdfe0010598800e3eec4fd6a909"
server_key_version: "encryptionkey@0"
I20260925 01:18:07.822819 31276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "4ee50451f0b34c66b99a0dfe2abb0e46"
format_stamp: "Formatted at 2026-09-25 01:18:07 on dist-test-slave-s12s"
server_key: "6421617b5e83ae79c09fb07b06412a3c"
server_key_iv: "2ca5dfdfe0010598800e3eec4fd6a909"
server_key_version: "encryptionkey@0"
I20260925 01:18:07.828065 31276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 01:18:07.831465 31291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:07.832944 31276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:18:07.833103 31276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "4ee50451f0b34c66b99a0dfe2abb0e46"
format_stamp: "Formatted at 2026-09-25 01:18:07 on dist-test-slave-s12s"
server_key: "6421617b5e83ae79c09fb07b06412a3c"
server_key_iv: "2ca5dfdfe0010598800e3eec4fd6a909"
server_key_version: "encryptionkey@0"
I20260925 01:18:07.833350 31276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:07.853524 31276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:07.857431 31276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:07.857730 31276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:07.858650 31276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:18:07.860210 31276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:18:07.860283 31276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:07.860426 31276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:18:07.860492 31276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:07.894047 31276 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:45797
I20260925 01:18:07.894116 31403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:45797 every 8 connection(s)
I20260925 01:18:07.895342 31276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:18:07.903983 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 31276
I20260925 01:18:07.904202 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:18:07.904876 20890 external_mini_cluster.cc:1428] Setting key 4e0b4b5174a98453eab59a512c6b0016
Sep 25 01:18:07 dist-test-slave-s12s krb5kdc[31170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299087, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:18:07.915647 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:07.897919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:59019 (local address 127.20.102.190:42697)
0925 01:18:07.898327 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:07.898331 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:07.899291 (+ 960us) server_negotiation.cc:407] Connection header received
0925 01:18:07.901132 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:07.901137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:07.901254 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:07.901378 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:07.902824 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:07.903607 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:07.905511 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:07.906061 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:07.906185 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:07.910038 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:07.910043 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:07.910046 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:07.910077 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:07.912548 (+ 2471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:07.913283 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:07.913288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:07.913292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:07.913419 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:07.913918 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:07.913930 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:07.913932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:07.914188 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:07.914358 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:07.915056 (+ 698us) server_negotiation.cc:299] Negotiation successful
0925 01:18:07.915288 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":205,"threads_started":1}
I20260925 01:18:07.916561 31407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:07.898468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:42697 (local address 127.20.102.129:59019)
0925 01:18:07.899151 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:07.899188 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:07.900866 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:07.901610 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:07.901619 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:07.902107 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:18:07.902655 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:07.902673 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:07.903766 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:07.903770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:07.905331 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:07.905344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:07.906274 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:07.906279 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:07.906474 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:07.907132 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:18:07.907158 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:18:07.909778 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:18:07.912727 (+ 2949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:18:07.912736 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:18:07.912750 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:18:07.913153 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:18:07.913565 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:18:07.913570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:18:07.913574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:18:07.913768 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:18:07.914320 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:18:07.914328 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:18:07.914669 (+ 341us) client_negotiation.cc:769] Sending connection context
0925 01:18:07.914930 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 01:18:07.915438 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":257,"threads_started":1}
I20260925 01:18:07.918028 31404 heartbeater.cc:347] Connected to a master server at 127.20.102.190:42697
I20260925 01:18:07.918342 31404 heartbeater.cc:464] Registering TS with master...
I20260925 01:18:07.919220 31404 heartbeater.cc:511] Master 127.20.102.190:42697 requested a full tablet report, sending...
I20260925 01:18:07.921451 31215 ts_manager.cc:194] Registered new tserver with Master: 4ee50451f0b34c66b99a0dfe2abb0e46 (127.20.102.129:45797)
I20260925 01:18:07.922890 31215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:59019
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:18:07.957782 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:42697
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:18:08.280467 31410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:08.280788 31410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:08.280951 31410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:08.287949 31410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:08.288038 31410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:08.288228 31410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:18:08.297498 31410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:08.298699 31410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:08.299958 31410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:08.314862 31415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:08.314848 31418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:08.314882 31416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:08.315696 31410 server_base.cc:1034] running on GCE node
I20260925 01:18:08.316313 31410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:08.317317 31410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:08.318562 31410 hybrid_clock.cc:648] HybridClock initialized: now 1790299088318515 us; error 61 us; skew 500 ppm
I20260925 01:18:08.318862 31410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:18:08 dist-test-slave-s12s krb5kdc[31170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299088, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:18:08.322081 31410 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:18:08.326176 31410 webserver.cc:460] Webserver started at http://127.20.102.130:37037/ using document root <none> and password file <none>
I20260925 01:18:08.326828 31410 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:08.326974 31410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:08.327266 31410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:18:08.330091 31410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "b93048dd3d7643d198c598c32c078149"
format_stamp: "Formatted at 2026-09-25 01:18:08 on dist-test-slave-s12s"
server_key: "8d54f0f10e92fccb724b12c7c354bda4"
server_key_iv: "90cc29268471f28d36291f5a74fb3044"
server_key_version: "encryptionkey@0"
I20260925 01:18:08.330933 31410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "b93048dd3d7643d198c598c32c078149"
format_stamp: "Formatted at 2026-09-25 01:18:08 on dist-test-slave-s12s"
server_key: "8d54f0f10e92fccb724b12c7c354bda4"
server_key_iv: "90cc29268471f28d36291f5a74fb3044"
server_key_version: "encryptionkey@0"
I20260925 01:18:08.335875 31410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:18:08.339331 31425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:08.340780 31410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:18:08.340940 31410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "b93048dd3d7643d198c598c32c078149"
format_stamp: "Formatted at 2026-09-25 01:18:08 on dist-test-slave-s12s"
server_key: "8d54f0f10e92fccb724b12c7c354bda4"
server_key_iv: "90cc29268471f28d36291f5a74fb3044"
server_key_version: "encryptionkey@0"
I20260925 01:18:08.341120 31410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:08.363111 31410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:08.366859 31410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:08.367149 31410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:08.368069 31410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:18:08.369385 31410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:18:08.369484 31410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:08.369652 31410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:18:08.369722 31410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:08.403278 31410 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:40249
I20260925 01:18:08.403331 31537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:40249 every 8 connection(s)
I20260925 01:18:08.404652 31410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:18:08.407245 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 31410
I20260925 01:18:08.407456 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:18:08.407925 20890 external_mini_cluster.cc:1428] Setting key a77edadb24b8d6e1586138ede97e978e
Sep 25 01:18:08 dist-test-slave-s12s krb5kdc[31170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299088, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:18:08.426707 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:08.407382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:47213 (local address 127.20.102.190:42697)
0925 01:18:08.407609 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:08.407616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:08.408936 (+ 1320us) server_negotiation.cc:407] Connection header received
0925 01:18:08.411252 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:08.411256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:08.411330 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:08.411412 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:08.413529 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.414229 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.416733 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.417158 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.417303 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:08.420853 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:08.420859 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:08.420862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:08.420895 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:08.423406 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:08.424118 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:08.424128 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:08.424130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:08.424254 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:08.424754 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:08.424762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:08.424771 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:08.425180 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:08.425349 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:08.426023 (+ 674us) server_negotiation.cc:299] Negotiation successful
0925 01:18:08.426281 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 01:18:08.427289 31540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:08.407846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:42697 (local address 127.20.102.130:47213)
0925 01:18:08.408723 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:08.408810 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:08.410950 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:08.411680 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:08.411694 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:08.412506 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:18:08.413348 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:08.413374 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.414379 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.414405 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:08.416542 (+ 2137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:08.416558 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.417379 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.417388 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:08.417696 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:08.418427 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:18:08.418462 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:18:08.420610 (+ 2148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:18:08.423553 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:18:08.423562 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:18:08.423577 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:18:08.423984 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:18:08.424365 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:18:08.424371 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:18:08.424375 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:18:08.424563 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:18:08.425307 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:18:08.425313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:18:08.425620 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 01:18:08.425844 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 01:18:08.426207 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":364,"threads_started":1}
I20260925 01:18:08.428908 31538 heartbeater.cc:347] Connected to a master server at 127.20.102.190:42697
I20260925 01:18:08.429265 31538 heartbeater.cc:464] Registering TS with master...
I20260925 01:18:08.430188 31538 heartbeater.cc:511] Master 127.20.102.190:42697 requested a full tablet report, sending...
I20260925 01:18:08.431862 31215 ts_manager.cc:194] Registered new tserver with Master: b93048dd3d7643d198c598c32c078149 (127.20.102.130:40249)
I20260925 01:18:08.432765 31215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:47213
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:18:08.463197 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:42697
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:40983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:18:08.786000 31543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:08.786306 31543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:08.786482 31543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:08.793579 31543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:08.793710 31543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:08.793888 31543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:18:08.803157 31543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:40983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:42697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:08.804458 31543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:08.805763 31543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:08.818596 31548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:08.818671 31551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:08.818636 31549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:08.819767 31543 server_base.cc:1034] running on GCE node
I20260925 01:18:08.820327 31543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:08.821306 31543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:08.822535 31543 hybrid_clock.cc:648] HybridClock initialized: now 1790299088822491 us; error 57 us; skew 500 ppm
I20260925 01:18:08.822830 31543 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:18:08 dist-test-slave-s12s krb5kdc[31170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299088, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:18:08.826092 31543 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:18:08.827368 31543 webserver.cc:460] Webserver started at http://127.20.102.131:43547/ using document root <none> and password file <none>
I20260925 01:18:08.828023 31543 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:08.828157 31543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:08.828460 31543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:18:08.831317 31543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "0b03a7c8a79140d2932cf092c8d449ec"
format_stamp: "Formatted at 2026-09-25 01:18:08 on dist-test-slave-s12s"
server_key: "855b0872fe1adbea9544a343dceb53a3"
server_key_iv: "c89a1f49cceaed092f3cdb00a3321422"
server_key_version: "encryptionkey@0"
I20260925 01:18:08.832118 31543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "0b03a7c8a79140d2932cf092c8d449ec"
format_stamp: "Formatted at 2026-09-25 01:18:08 on dist-test-slave-s12s"
server_key: "855b0872fe1adbea9544a343dceb53a3"
server_key_iv: "c89a1f49cceaed092f3cdb00a3321422"
server_key_version: "encryptionkey@0"
I20260925 01:18:08.837344 31543 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260925 01:18:08.840819 31558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:08.842082 31543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:18:08.842267 31543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "0b03a7c8a79140d2932cf092c8d449ec"
format_stamp: "Formatted at 2026-09-25 01:18:08 on dist-test-slave-s12s"
server_key: "855b0872fe1adbea9544a343dceb53a3"
server_key_iv: "c89a1f49cceaed092f3cdb00a3321422"
server_key_version: "encryptionkey@0"
I20260925 01:18:08.842486 31543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:08.863406 31543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:08.867762 31543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:08.868069 31543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:08.869063 31543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:18:08.870357 31543 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:18:08.870431 31543 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:08.870606 31543 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:18:08.870668 31543 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:08.904207 31543 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:46399
I20260925 01:18:08.904299 31670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:46399 every 8 connection(s)
I20260925 01:18:08.905532 31543 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:18:08.913273 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 31543
I20260925 01:18:08.913468 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:18:08.913889 20890 external_mini_cluster.cc:1428] Setting key af712258d430f1c0bf6e8969f6c17989
Sep 25 01:18:08 dist-test-slave-s12s krb5kdc[31170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299088, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:18:08.924573 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:08.908132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:35313 (local address 127.20.102.190:42697)
0925 01:18:08.908347 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:08.908351 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:08.909512 (+ 1161us) server_negotiation.cc:407] Connection header received
0925 01:18:08.911288 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:08.911293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:08.911401 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:08.911520 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:08.913029 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.913883 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.915671 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.916166 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.916266 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:08.919706 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:08.919711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:08.919714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:08.919745 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:08.922107 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:08.922772 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:08.922775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:08.922778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:08.922897 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:08.923249 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:08.923252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:08.923254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:08.923495 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:08.923600 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:08.924194 (+ 594us) server_negotiation.cc:299] Negotiation successful
0925 01:18:08.924343 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:18:08.925437 31673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:08.908579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:42697 (local address 127.20.102.131:35313)
0925 01:18:08.909326 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:08.909363 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:08.911035 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:08.911752 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:08.911760 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:08.912259 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:18:08.912861 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:08.912879 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.914040 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.914044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:08.915505 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:08.915520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.916389 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.916396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:08.916756 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:08.917458 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:18:08.917486 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:18:08.919530 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:18:08.922235 (+ 2705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:18:08.922241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:18:08.922252 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:18:08.922665 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:18:08.922988 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:18:08.922991 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:18:08.922993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:18:08.923152 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:18:08.923594 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:18:08.923599 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:18:08.923874 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 01:18:08.924078 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 01:18:08.924412 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":273,"threads_started":1}
I20260925 01:18:08.925809 31404 heartbeater.cc:503] Master 127.20.102.190:42697 was elected leader, sending a full tablet report...
I20260925 01:18:08.926877 31671 heartbeater.cc:347] Connected to a master server at 127.20.102.190:42697
I20260925 01:18:08.927196 31671 heartbeater.cc:464] Registering TS with master...
I20260925 01:18:08.928148 31671 heartbeater.cc:511] Master 127.20.102.190:42697 requested a full tablet report, sending...
I20260925 01:18:08.929483 31215 ts_manager.cc:194] Registered new tserver with Master: 0b03a7c8a79140d2932cf092c8d449ec (127.20.102.131:46399)
I20260925 01:18:08.930239 31215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:35313
I20260925 01:18:08.941735 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:18:08.962936 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:08.950651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.20.102.190:42697)
0925 01:18:08.950813 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:08.950818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:08.951321 (+ 503us) server_negotiation.cc:407] Connection header received
0925 01:18:08.951473 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:08.951481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:08.951550 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:08.951642 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:08.953053 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.953832 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.955053 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.955526 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.955633 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:08.957858 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:08.957863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:08.957866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:08.957896 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:08.960578 (+ 2682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:08.961292 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:08.961296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:08.961299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:08.961430 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:08.961761 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:08.961764 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:08.961766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:08.962074 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:08.962256 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:08.962564 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 01:18:08.962716 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:18:08.967000 31215 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:18:08.973770 31215 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:18:08.998498 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:08.986175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.20.102.131:46399)
0925 01:18:08.987707 (+ 1532us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:08.987716 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:08.988908 (+ 1192us) server_negotiation.cc:407] Connection header received
0925 01:18:08.989326 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:08.989339 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:08.989626 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:08.989800 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:18:08.990879 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.992585 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.995473 (+ 2888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.996918 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.997085 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:08.997287 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:08.997966 (+ 679us) server_negotiation.cc:299] Negotiation successful
0925 01:18:08.998155 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1358,"thread_start_us":1205,"threads_started":1}
I20260925 01:18:08.999622 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:08.985871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:46399 (local address 127.0.0.1:51640)
0925 01:18:08.988736 (+ 2865us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:08.988798 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:08.989070 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:08.990171 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:08.990178 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:08.990209 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:18:08.990659 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:08.990673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.992791 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.992797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:08.995283 (+ 2486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:08.995303 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.997473 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.997479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:08.997681 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:08.997745 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 01:18:08.998983 (+ 1238us) client_negotiation.cc:240] Negotiation successful
0925 01:18:08.999174 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2576,"mutex_wait_us":1,"spinlock_wait_cycles":11008,"thread_start_us":2545,"threads_started":1}
I20260925 01:18:09.000937 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:08.986833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.20.102.129:45797)
0925 01:18:08.988230 (+ 1397us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:08.988241 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:08.988903 (+ 662us) server_negotiation.cc:407] Connection header received
0925 01:18:08.990420 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:08.990428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:08.990667 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:08.990824 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:18:08.991841 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.994465 (+ 2624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.997998 (+ 3533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.999416 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.999593 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:08.999792 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:09.000437 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 01:18:09.000613 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1213,"thread_start_us":1035,"threads_started":1}
I20260925 01:18:09.002811 31683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:08.986152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:45797 (local address 127.0.0.1:53744)
0925 01:18:08.988736 (+ 2584us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:08.988798 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:08.989070 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:08.991167 (+ 2097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:08.991172 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:08.991196 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:18:08.991630 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:08.991643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.994644 (+ 3001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:08.994649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:08.996825 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:08.996844 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:08.999993 (+ 3149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.000000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:09.000204 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:09.000248 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 01:18:09.001145 (+ 897us) client_negotiation.cc:240] Negotiation successful
0925 01:18:09.001275 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2444,"mutex_wait_us":1,"spinlock_wait_cycles":2816,"thread_start_us":2113,"threads_started":1}
I20260925 01:18:09.004901 31606 tablet_service.cc:1467] Processing CreateTablet for tablet b8772e5ea6d048c6a8814045cddd9d24 (DEFAULT_TABLE table=test-table [id=67a13117a997476aae3c75dbaa7c9377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:18:09.006393 31606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8772e5ea6d048c6a8814045cddd9d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:18:09.008988 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:08.989231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:40249 (local address 127.0.0.1:58602)
0925 01:18:09.001358 (+ 12127us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:09.001383 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:09.001487 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:09.003050 (+ 1563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:09.003055 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:09.003079 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:18:09.003232 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:09.003239 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.004650 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.004657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:09.006598 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:09.006610 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.008308 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.008313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:09.008510 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:09.008548 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 01:18:09.008634 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 01:18:09.008790 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12037,"thread_start_us":15569,"threads_started":1}
I20260925 01:18:09.008795 31339 tablet_service.cc:1467] Processing CreateTablet for tablet b8772e5ea6d048c6a8814045cddd9d24 (DEFAULT_TABLE table=test-table [id=67a13117a997476aae3c75dbaa7c9377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:18:09.009697 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:08.989472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.20.102.130:40249)
0925 01:18:09.002242 (+ 12770us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:09.002251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:09.002281 (+ 30us) server_negotiation.cc:407] Connection header received
0925 01:18:09.002394 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:09.002401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:09.002630 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:09.003369 (+ 739us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:18:09.003564 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.004495 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.006762 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.007848 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.008024 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:09.008204 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:09.009227 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0925 01:18:09.009388 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12600,"thread_start_us":13282,"threads_started":1}
I20260925 01:18:09.010223 31339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8772e5ea6d048c6a8814045cddd9d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:18:09.012945 31473 tablet_service.cc:1467] Processing CreateTablet for tablet b8772e5ea6d048c6a8814045cddd9d24 (DEFAULT_TABLE table=test-table [id=67a13117a997476aae3c75dbaa7c9377]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:18:09.014053 31473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8772e5ea6d048c6a8814045cddd9d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:18:09.025488 31689 tablet_bootstrap.cc:492] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec: Bootstrap starting.
I20260925 01:18:09.030467 31689 tablet_bootstrap.cc:654] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec: Neither blocks nor log segments found. Creating new log.
I20260925 01:18:09.031338 31691 tablet_bootstrap.cc:492] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46: Bootstrap starting.
I20260925 01:18:09.032051 31689 log.cc:824] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec: Log is configured to *not* fsync() on all Append() calls
I20260925 01:18:09.032997 31690 tablet_bootstrap.cc:492] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149: Bootstrap starting.
I20260925 01:18:09.034971 31691 tablet_bootstrap.cc:654] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46: Neither blocks nor log segments found. Creating new log.
I20260925 01:18:09.035977 31689 tablet_bootstrap.cc:492] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec: No bootstrap required, opened a new log
I20260925 01:18:09.036231 31689 ts_tablet_manager.cc:1397] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260925 01:18:09.036285 31691 log.cc:824] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46: Log is configured to *not* fsync() on all Append() calls
I20260925 01:18:09.036811 31690 tablet_bootstrap.cc:654] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149: Neither blocks nor log segments found. Creating new log.
I20260925 01:18:09.037937 31690 log.cc:824] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149: Log is configured to *not* fsync() on all Append() calls
I20260925 01:18:09.039506 31691 tablet_bootstrap.cc:492] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46: No bootstrap required, opened a new log
I20260925 01:18:09.039773 31691 ts_tablet_manager.cc:1397] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 01:18:09.041016 31690 tablet_bootstrap.cc:492] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149: No bootstrap required, opened a new log
I20260925 01:18:09.041313 31690 ts_tablet_manager.cc:1397] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260925 01:18:09.044514 31689 raft_consensus.cc:348] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b03a7c8a79140d2932cf092c8d449ec" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46399 } } peers { permanent_uuid: "4ee50451f0b34c66b99a0dfe2abb0e46" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45797 } } peers { permanent_uuid: "b93048dd3d7643d198c598c32c078149" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40249 } }
I20260925 01:18:09.045089 31689 raft_consensus.cc:374] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:18:09.045208 31689 raft_consensus.cc:729] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b03a7c8a79140d2932cf092c8d449ec, State: Initialized, Role: FOLLOWER
I20260925 01:18:09.045833 31689 consensus_queue.cc:260] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b03a7c8a79140d2932cf092c8d449ec" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46399 } } peers { permanent_uuid: "4ee50451f0b34c66b99a0dfe2abb0e46" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45797 } } peers { permanent_uuid: "b93048dd3d7643d198c598c32c078149" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40249 } }
I20260925 01:18:09.049396 31671 heartbeater.cc:503] Master 127.20.102.190:42697 was elected leader, sending a full tablet report...
I20260925 01:18:09.050110 31689 ts_tablet_manager.cc:1428] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260925 01:18:09.050604 31691 raft_consensus.cc:348] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b03a7c8a79140d2932cf092c8d449ec" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46399 } } peers { permanent_uuid: "4ee50451f0b34c66b99a0dfe2abb0e46" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45797 } } peers { permanent_uuid: "b93048dd3d7643d198c598c32c078149" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40249 } }
I20260925 01:18:09.051021 31691 raft_consensus.cc:374] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:18:09.051118 31691 raft_consensus.cc:729] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee50451f0b34c66b99a0dfe2abb0e46, State: Initialized, Role: FOLLOWER
I20260925 01:18:09.051712 31691 consensus_queue.cc:260] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b03a7c8a79140d2932cf092c8d449ec" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46399 } } peers { permanent_uuid: "4ee50451f0b34c66b99a0dfe2abb0e46" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45797 } } peers { permanent_uuid: "b93048dd3d7643d198c598c32c078149" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40249 } }
I20260925 01:18:09.051805 31690 raft_consensus.cc:348] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b03a7c8a79140d2932cf092c8d449ec" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46399 } } peers { permanent_uuid: "4ee50451f0b34c66b99a0dfe2abb0e46" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45797 } } peers { permanent_uuid: "b93048dd3d7643d198c598c32c078149" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40249 } }
I20260925 01:18:09.052271 31690 raft_consensus.cc:374] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:18:09.052399 31690 raft_consensus.cc:729] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b93048dd3d7643d198c598c32c078149, State: Initialized, Role: FOLLOWER
I20260925 01:18:09.053117 31690 consensus_queue.cc:260] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b03a7c8a79140d2932cf092c8d449ec" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46399 } } peers { permanent_uuid: "4ee50451f0b34c66b99a0dfe2abb0e46" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45797 } } peers { permanent_uuid: "b93048dd3d7643d198c598c32c078149" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40249 } }
I20260925 01:18:09.054332 31538 heartbeater.cc:503] Master 127.20.102.190:42697 was elected leader, sending a full tablet report...
I20260925 01:18:09.055243 31690 ts_tablet_manager.cc:1428] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260925 01:18:09.056864 31691 ts_tablet_manager.cc:1428] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46: Time spent starting tablet: real 0.017s user 0.010s sys 0.008s
W20260925 01:18:09.147647 31405 tablet.cc:2369] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:18:09.156585 31539 tablet.cc:2369] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:18:09.157084 31672 tablet.cc:2369] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:18:09.161564 31697 raft_consensus.cc:482] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:18:09.161850 31697 raft_consensus.cc:504] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b03a7c8a79140d2932cf092c8d449ec" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46399 } } peers { permanent_uuid: "4ee50451f0b34c66b99a0dfe2abb0e46" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45797 } } peers { permanent_uuid: "b93048dd3d7643d198c598c32c078149" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40249 } }
I20260925 01:18:09.163450 31697 leader_election.cc:288] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b03a7c8a79140d2932cf092c8d449ec (127.20.102.131:46399), 4ee50451f0b34c66b99a0dfe2abb0e46 (127.20.102.129:45797)
I20260925 01:18:09.172560 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.163901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:35397 (local address 127.20.102.129:45797)
0925 01:18:09.164124 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:09.164129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:09.164912 (+ 783us) server_negotiation.cc:407] Connection header received
0925 01:18:09.165049 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:09.165065 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:09.165136 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:09.165236 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:18:09.166096 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.167196 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.170345 (+ 3149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.171543 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.171737 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:09.171819 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:09.172130 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 01:18:09.172272 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 01:18:09.172562 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.164015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:42231 (local address 127.20.102.131:46399)
0925 01:18:09.164200 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:09.164208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:09.164839 (+ 631us) server_negotiation.cc:407] Connection header received
0925 01:18:09.165049 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:09.165056 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:09.165136 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:09.165236 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:18:09.166096 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.167197 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.170299 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.171572 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.171759 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:09.171841 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:09.172153 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 01:18:09.172279 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 01:18:09.172746 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.163733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:46399 (local address 127.20.102.130:42231)
0925 01:18:09.164719 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:09.164740 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:09.164910 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:09.165269 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:09.165274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:09.165312 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:18:09.165954 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:09.165967 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.167335 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.167340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:09.170109 (+ 2769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:09.170126 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.171810 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.171817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:09.171985 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:09.172009 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:18:09.172125 (+ 116us) client_negotiation.cc:240] Negotiation successful
0925 01:18:09.172279 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":218,"thread_start_us":644,"threads_started":1}
I20260925 01:18:09.172796 31698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.163852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:45797 (local address 127.20.102.130:35397)
0925 01:18:09.164813 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:09.164828 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:09.164921 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:09.165269 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:09.165274 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:09.165369 (+ 95us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:18:09.165862 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:09.165883 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.167337 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.167340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:09.170143 (+ 2803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:09.170153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.171779 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.171786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:09.171963 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:09.171988 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 01:18:09.172103 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 01:18:09.172272 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":199,"spinlock_wait_cycles":22272,"thread_start_us":453,"threads_started":1}
I20260925 01:18:09.173794 31626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8772e5ea6d048c6a8814045cddd9d24" candidate_uuid: "b93048dd3d7643d198c598c32c078149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b03a7c8a79140d2932cf092c8d449ec" is_pre_election: true
I20260925 01:18:09.173934 31359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8772e5ea6d048c6a8814045cddd9d24" candidate_uuid: "b93048dd3d7643d198c598c32c078149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee50451f0b34c66b99a0dfe2abb0e46" is_pre_election: true
I20260925 01:18:09.174369 31626 raft_consensus.cc:2454] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b93048dd3d7643d198c598c32c078149 in term 0.
I20260925 01:18:09.174484 31359 raft_consensus.cc:2454] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b93048dd3d7643d198c598c32c078149 in term 0.
I20260925 01:18:09.175136 31427 leader_election.cc:302] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ee50451f0b34c66b99a0dfe2abb0e46, b93048dd3d7643d198c598c32c078149; no voters:
I20260925 01:18:09.175550 31697 raft_consensus.cc:2781] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:18:09.175699 31697 raft_consensus.cc:482] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:18:09.175813 31697 raft_consensus.cc:3037] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:18:09.178007 31697 raft_consensus.cc:504] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b03a7c8a79140d2932cf092c8d449ec" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46399 } } peers { permanent_uuid: "4ee50451f0b34c66b99a0dfe2abb0e46" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45797 } } peers { permanent_uuid: "b93048dd3d7643d198c598c32c078149" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40249 } }
I20260925 01:18:09.178551 31697 leader_election.cc:288] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [CANDIDATE]: Term 1 election: Requested vote from peers 0b03a7c8a79140d2932cf092c8d449ec (127.20.102.131:46399), 4ee50451f0b34c66b99a0dfe2abb0e46 (127.20.102.129:45797)
I20260925 01:18:09.178915 31626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8772e5ea6d048c6a8814045cddd9d24" candidate_uuid: "b93048dd3d7643d198c598c32c078149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b03a7c8a79140d2932cf092c8d449ec"
I20260925 01:18:09.178979 31359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8772e5ea6d048c6a8814045cddd9d24" candidate_uuid: "b93048dd3d7643d198c598c32c078149" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee50451f0b34c66b99a0dfe2abb0e46"
I20260925 01:18:09.179107 31626 raft_consensus.cc:3037] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:18:09.179175 31359 raft_consensus.cc:3037] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:18:09.181308 31626 raft_consensus.cc:2454] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b93048dd3d7643d198c598c32c078149 in term 1.
I20260925 01:18:09.181375 31359 raft_consensus.cc:2454] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b93048dd3d7643d198c598c32c078149 in term 1.
I20260925 01:18:09.181774 31428 leader_election.cc:302] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b03a7c8a79140d2932cf092c8d449ec, b93048dd3d7643d198c598c32c078149; no voters:
I20260925 01:18:09.182085 31697 raft_consensus.cc:2781] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:18:09.182479 31697 raft_consensus.cc:686] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [term 1 LEADER]: Becoming Leader. State: Replica: b93048dd3d7643d198c598c32c078149, State: Running, Role: LEADER
I20260925 01:18:09.182916 31697 consensus_queue.cc:237] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b03a7c8a79140d2932cf092c8d449ec" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46399 } } peers { permanent_uuid: "4ee50451f0b34c66b99a0dfe2abb0e46" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45797 } } peers { permanent_uuid: "b93048dd3d7643d198c598c32c078149" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40249 } }
I20260925 01:18:09.187222 31215 catalog_manager.cc:5495] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 reported cstate change: term changed from 0 to 1, leader changed from <none> to b93048dd3d7643d198c598c32c078149 (127.20.102.130). New cstate: current_term: 1 leader_uuid: "b93048dd3d7643d198c598c32c078149" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b03a7c8a79140d2932cf092c8d449ec" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ee50451f0b34c66b99a0dfe2abb0e46" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b93048dd3d7643d198c598c32c078149" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40249 } health_report { overall_health: HEALTHY } } }
I20260925 01:18:09.232548 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.227374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.20.102.130:40249)
0925 01:18:09.227606 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:09.227610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:09.227628 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:18:09.227772 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:09.227775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:09.227852 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:09.227973 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:18:09.228334 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.229132 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.230375 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.230756 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.230844 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:09.231327 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:18:09.231846 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:18:09.231990 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:09.232103 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 01:18:09.232175 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 01:18:09.240898 31359 raft_consensus.cc:1261] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46 [term 1 FOLLOWER]: Refusing update from remote peer b93048dd3d7643d198c598c32c078149: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:18:09.240927 31626 raft_consensus.cc:1261] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec [term 1 FOLLOWER]: Refusing update from remote peer b93048dd3d7643d198c598c32c078149: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:18:09.241838 31700 consensus_queue.cc:1035] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee50451f0b34c66b99a0dfe2abb0e46" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 45797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:18:09.242213 31697 consensus_queue.cc:1035] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b03a7c8a79140d2932cf092c8d449ec" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:18:09.262764 31705 mvcc.cc:205] Tried to move back new op lower bound from 7333065069520756736 to 7333065069296893952. Current Snapshot: MvccSnapshot[applied={T|T < 7333065069520756736}]
I20260925 01:18:09.279796 31215 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:18:09.280014 31215 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:18:09.283588 31215 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f51024c15eb4999b2f2a1070b16b7b1: Sending DeleteTablet for 3 replicas of tablet b8772e5ea6d048c6a8814045cddd9d24
I20260925 01:18:09.284605 31606 tablet_service.cc:1514] Processing DeleteTablet for tablet b8772e5ea6d048c6a8814045cddd9d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:18:09 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:51640
I20260925 01:18:09.285202 31339 tablet_service.cc:1514] Processing DeleteTablet for tablet b8772e5ea6d048c6a8814045cddd9d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:18:09 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:53744
I20260925 01:18:09.285549 31473 tablet_service.cc:1514] Processing DeleteTablet for tablet b8772e5ea6d048c6a8814045cddd9d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:18:09 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:58602
I20260925 01:18:09.286692 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 31276
I20260925 01:18:09.287014 31713 tablet_replica.cc:330] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec: stopping tablet replica
I20260925 01:18:09.287468 31713 raft_consensus.cc:2229] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:18:09.288051 31713 raft_consensus.cc:2258] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:18:09.290282 31713 ts_tablet_manager.cc:1905] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:18:09.304111 31715 tablet_replica.cc:330] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149: stopping tablet replica
I20260925 01:18:09.304648 31715 raft_consensus.cc:2229] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:18:09.305159 31714 tablet_replica.cc:330] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46: stopping tablet replica
I20260925 01:18:09.305521 31715 raft_consensus.cc:2258] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:18:09.305593 31714 raft_consensus.cc:2229] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:18:09.306125 31714 raft_consensus.cc:2258] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:18:09.307886 31715 ts_tablet_manager.cc:1905] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 01:18:09 dist-test-slave-s12s krb5kdc[31170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:18:09.312347 31713 ts_tablet_manager.cc:1918] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:18:09.312531 31713 log.cc:1196] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/b8772e5ea6d048c6a8814045cddd9d24
I20260925 01:18:09.313153 31713 ts_tablet_manager.cc:1939] T b8772e5ea6d048c6a8814045cddd9d24 P 0b03a7c8a79140d2932cf092c8d449ec: Deleting consensus metadata
I20260925 01:18:09.314366 31714 ts_tablet_manager.cc:1905] T b8772e5ea6d048c6a8814045cddd9d24 P 4ee50451f0b34c66b99a0dfe2abb0e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:18:09.315074 31201 catalog_manager.cc:4841] TS 0b03a7c8a79140d2932cf092c8d449ec (127.20.102.131:46399): tablet b8772e5ea6d048c6a8814045cddd9d24 (table test-table [id=67a13117a997476aae3c75dbaa7c9377]) successfully deleted
I20260925 01:18:09.315909 31715 ts_tablet_manager.cc:1918] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:18:09.316099 31715 log.cc:1196] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/b8772e5ea6d048c6a8814045cddd9d24
I20260925 01:18:09.316707 31715 ts_tablet_manager.cc:1939] T b8772e5ea6d048c6a8814045cddd9d24 P b93048dd3d7643d198c598c32c078149: Deleting consensus metadata
I20260925 01:18:09.318454 31200 catalog_manager.cc:4841] TS b93048dd3d7643d198c598c32c078149 (127.20.102.130:40249): tablet b8772e5ea6d048c6a8814045cddd9d24 (table test-table [id=67a13117a997476aae3c75dbaa7c9377]) successfully deleted
I20260925 01:18:09.323273 31686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.288550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.20.102.129:45797)
0925 01:18:09.288804 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:09.288813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:09.294146 (+ 5333us) server_negotiation.cc:407] Connection header received
0925 01:18:09.295002 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:09.295012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:09.295127 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:09.295246 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:09.299435 (+ 4189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.301013 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.306610 (+ 5597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.307208 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.307313 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:09.316665 (+ 9352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:09.316708 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:09.316727 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:09.316773 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:09.319896 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:09.320575 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:09.320584 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:09.320587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:09.320791 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:09.321263 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:09.321268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:09.321272 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:09.321734 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:09.322036 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:09.322646 (+ 610us) server_negotiation.cc:299] Negotiation successful
0925 01:18:09.322882 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:18:09.324985 31399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:09.414978 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 31276
W20260925 01:18:09.433728 31200 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.102.129:45797 (error 108)
I20260925 01:18:09.434605 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 31410
I20260925 01:18:09.441286 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.440771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:45797 (local address 127.0.0.1:53758)
0925 01:18:09.440980 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:09.441105 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 01:18:09.441581 31200 catalog_manager.cc:4584] TS 4ee50451f0b34c66b99a0dfe2abb0e46 (127.20.102.129:45797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b8772e5ea6d048c6a8814045cddd9d24: Network error: Client connection negotiation failed: client connection to 127.20.102.129:45797: connect: Connection refused (error 111)
Sep 25 01:18:09 dist-test-slave-s12s krb5kdc[31170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:18:09.453794 31688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.436738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58620 (local address 127.20.102.130:40249)
0925 01:18:09.436969 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:09.436978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:09.437001 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:18:09.437083 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:09.437091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:09.437235 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:09.437369 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:09.439804 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.441086 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.442504 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.442945 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.443085 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:09.447705 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:09.447726 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:09.447740 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:09.447784 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:09.450980 (+ 3196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:09.451660 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:09.451674 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:09.451677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:09.451830 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:09.452220 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:09.452225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:09.452229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:09.452623 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:09.452980 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:09.453213 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 01:18:09.453449 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 01:18:09.454475 31533 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:09.495234 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.494749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:45797 (local address 127.0.0.1:53768)
0925 01:18:09.494968 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:09.495068 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 01:18:09.496295 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.495936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:45797 (local address 127.0.0.1:53784)
0925 01:18:09.496108 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:09.496172 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 01:18:09.556797 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.556268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:45797 (local address 127.0.0.1:53794)
0925 01:18:09.556483 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:09.556582 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 01:18:09.557807 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.557466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:45797 (local address 127.0.0.1:53796)
0925 01:18:09.557620 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:09.557693 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 01:18:09.568943 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 31410
I20260925 01:18:09.587827 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 31543
Sep 25 01:18:09 dist-test-slave-s12s krb5kdc[31170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:18:09.602942 31684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.588606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51642 (local address 127.20.102.131:46399)
0925 01:18:09.588824 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:09.588829 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:09.588848 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:18:09.589026 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:09.589031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:09.589097 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:09.589180 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:09.590547 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.591300 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.592566 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:09.593058 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:09.593160 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:09.597447 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:09.597462 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:09.597479 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:09.597544 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:09.600282 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:09.601095 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:09.601104 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:09.601106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:09.601251 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:09.601591 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:09.601594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:09.601596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:09.601935 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:09.602178 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:09.602408 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 01:18:09.602582 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 01:18:09.603618 31666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:09.638888 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.638454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:45797 (local address 127.0.0.1:53800)
0925 01:18:09.638663 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:09.638744 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 01:18:09.640038 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.639632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:45797 (local address 127.0.0.1:53804)
0925 01:18:09.639845 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:09.639911 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 01:18:09.692572 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 31543
I20260925 01:18:09.709640 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 31183
I20260925 01:18:09.710814 31245 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:09.819888 31685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:09.819196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:45797 (local address 127.0.0.1:53818)
0925 01:18:09.819519 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:09.819705 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:45797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260925 01:18:09.819904 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 31183
2026-09-25T01:18:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4291 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:18:09 dist-test-slave-s12s krb5kdc[31724](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:18:09 dist-test-slave-s12s krb5kdc[31724](info): set up 2 sockets
Sep 25 01:18:09 dist-test-slave-s12s krb5kdc[31724](info): commencing operation
krb5kdc: starting...
W20260925 01:18:10.921813 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:18:11 dist-test-slave-s12s krb5kdc[31724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:18:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:18:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/rangeradmin_127.20.102.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/rangeradmin_127.20.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/rangerlookup_127.20.102.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/rangerlookup_127.20.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/HTTP_127.20.102.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/HTTP_127.20.102.148@KRBTEST.COM.keytab.
I20260925 01:18:11.245889 20890 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task_TC7zy/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/postgres -l logfile start
2026-09-25 01:18:13.690 UTC [31754] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 01:18:13.690 UTC [31754] LOG: listening on IPv4 address "127.20.102.148", port 41209
2026-09-25 01:18:13.693 UTC [31754] LOG: listening on Unix socket "/tmp/.s.PGSQL.41209"
2026-09-25 01:18:13.716 UTC [31757] LOG: database system was shut down at 2026-09-25 01:18:12 UTC
2026-09-25 01:18:13.720 UTC [31754] LOG: database system is ready to accept connections
I20260925 01:18:14.701347 20890 mini_postgres.cc:96] Postgres bound to 41209
2026-09-25 01:18:14.721 UTC [31765] FATAL: database "slave" does not exist
127.20.102.148:41209 - accepting connections
I20260925 01:18:14.722051 20890 mini_ranger.cc:161] Starting Ranger...
I20260925 01:18:14.751619 20890 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 01:18:15.204923 20890 mini_ranger.cc:87] Created ranger Postgres database
I20260925 01:18:15.205065 20890 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-admin
2026-09-25 01:18:15,488 [I] DB FLAVOR :POSTGRES
2026-09-25 01:18:15,488 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 01:18:15,488 [I] Checking connection..
2026-09-25 01:18:15,488 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 01:18:15,748 [I] Checking connection passed.
2026-09-25 01:18:15,749 [I] --------- Verifying version history table ---------
2026-09-25 01:18:15,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:18:15,996 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 01:18:15,997 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:18:16,234 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 01:18:16,234 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 01:18:16,235 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 01:18:16.463 UTC [31829] WARNING: there is no transaction in progress
2026-09-25 01:18:16,469 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 01:18:16,470 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:18:16,730 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 01:18:16,730 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 01:18:16,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 01:18:16,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 01:18:17,236 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s12s.c.gcp-upstream.internal','N') ;"
2026-09-25 01:18:17,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 01:18:17,734 [I] Table x_portal_user does not exist in database ranger
2026-09-25 01:18:17,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 01:18:17,979 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 01:18:17,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 01:18:18,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 01:18:18,460 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 01:18:18,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 01:18:19.402 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.418 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.433 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.447 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.463 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.482 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.544 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.551 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.564 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.574 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.584 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.597 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.607 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.613 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.620 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:19.627 UTC [31937] WARNING: there is no transaction in progress
2026-09-25 01:18:20,237 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 01:18:20,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 01:18:20,499 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 01:18:20,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 01:18:20,728 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 01:18:20,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 01:18:20,971 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 01:18:20,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 01:18:21,226 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 01:18:21,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s12s.c.gcp-upstream.internal';"
2026-09-25 01:18:21,743 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 01:18:21,744 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 01:18:22,001 [I] DB_PATCHES have already been applied
I20260925 01:18:22.007320 20890 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-admin:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task_TC7zy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 01:18:22.007439 20890 mini_ranger.cc:193] Using host: 127.20.102.148
I20260925 01:18:22.022810 20890 mini_ranger.cc:227] Ranger admin URL: http://127.20.102.148:60843
Sep 25, 2026 1:18:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:18:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:18:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 1:18:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 1:18:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 1:18:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 1:18:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 01:18:23.143790 20890 mini_ranger.cc:160] Time spent starting Ranger: real 8.422s user 0.003s sys 0.037s
Sep 25, 2026 1:18:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 1:18:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 1:18:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 1:18:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 1:18:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 1:18:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 1:18:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 1:18:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 1:18:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 1:18:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 1:18:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 1:18:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 1:18:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 1:18:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 1:18:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 1:18:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 1:18:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 1:18:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 01:18:44.212088 20890 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/rangerkms_127.20.102.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/rangerkms_127.20.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.102.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.102.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.102.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/HTTP_127.20.102.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.102.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/HTTP_127.20.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 01:18:44 dist-test-slave-s12s krb5kdc[31724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299124, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 01:18:44.388579 20890 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 01:18:44.414355 20890 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 01:18:45.200254 20890 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 01:18:45.200428 20890 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-kms
I20260925 01:18:45.200603 20890 mini_ranger_kms.cc:227] Using postgres at 127.20.102.148:41209
2026-09-25 01:18:46,286 [I] DB FLAVOR :POSTGRES
2026-09-25 01:18:46,286 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 01:18:46,287 [I] Checking connection
2026-09-25 01:18:46,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 01:18:46,519 [I] connection success
2026-09-25 01:18:46,519 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 01:18:46,520 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 01:18:46,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 01:18:46,754 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 01:18:46,754 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 01:18:46,754 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 01:18:46,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:41209/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 01:18:47,028 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 01:18:47.471194 20890 mini_ranger_kms.cc:310] Created kms service
I20260925 01:18:47.674882 20890 mini_ranger_kms.cc:326] Created kudu user
I20260925 01:18:47.720758 20890 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 01:18:47 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299124, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:18:48.082513 20890 mini_ranger_kms.cc:384] Added ranger policy
I20260925 01:18:48.082863 20890 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-kms:/tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 01:18:48.082919 20890 mini_ranger_kms.cc:240] Using host: 127.20.102.148
I20260925 01:18:48.102361 20890 mini_ranger_kms.cc:276] Ranger KMS PID: 32117
I20260925 01:18:48.102571 20890 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.102.148:49467
Sep 25, 2026 1:18:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:18:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:18:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 1:18:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 1:18:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 1:18:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 01:18:50.419895 20890 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.031s user 0.149s sys 0.386s
I20260925 01:18:50.420184 20890 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 01:18:50.420413 20890 mini_ranger_kms.cc:401] 127.20.102.148:49467/kms/v1/keys
Sep 25 01:18:50 dist-test-slave-s12s krb5kdc[31724](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299130, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.102.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 01:18:51 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299130, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.102.148@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
Sep 25 01:18:51 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299130, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.102.148@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
Sep 25 01:18:53 dist-test-slave-s12s krb5kdc[31724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:18:53.840772 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:44231
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:44231
--ranger_config_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.102.148:49467/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:18:54.187745 32168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:54.188025 32168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:54.188158 32168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:54.195669 32168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:18:54.195776 32168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:54.195847 32168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:54.195941 32168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:18:54.196007 32168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:18:54.206550 32168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.102.148:49467/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:44231
--ranger_config_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:44231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:54.207883 32168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:54.209307 32168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:54.221669 32176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:54.221756 32173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:54.221732 32174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:54.222958 32168 server_base.cc:1034] running on GCE node
I20260925 01:18:54.223816 32168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:54.225394 32168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:54.226622 32168 hybrid_clock.cc:648] HybridClock initialized: now 1790299134226571 us; error 45 us; skew 500 ppm
I20260925 01:18:54.226974 32168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:18:54 dist-test-slave-s12s krb5kdc[31724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299134, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:18:54.236748 32168 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:18:54.238420 32168 webserver.cc:460] Webserver started at http://127.20.102.190:33151/ using document root <none> and password file <none>
I20260925 01:18:54.239125 32168 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:54.239259 32168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:54.239570 32168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:18:54 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299134, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:18:54.298110 32168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "d7dac9f4f3ab4a51a0c041ac2b57bee0"
format_stamp: "Formatted at 2026-09-25 01:18:54 on dist-test-slave-s12s"
server_key: "97f43ea85b2acdaa5fefe6c9db8a4900"
server_key_iv: "f579151df50042121898a3ed26d178bd"
server_key_version: "kuduclusterkey@0"
I20260925 01:18:54.299211 32168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "d7dac9f4f3ab4a51a0c041ac2b57bee0"
format_stamp: "Formatted at 2026-09-25 01:18:54 on dist-test-slave-s12s"
server_key: "97f43ea85b2acdaa5fefe6c9db8a4900"
server_key_iv: "f579151df50042121898a3ed26d178bd"
server_key_version: "kuduclusterkey@0"
I20260925 01:18:54.306620 32168 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260925 01:18:54.326694 32183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:54.328438 32168 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260925 01:18:54.328653 32168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "d7dac9f4f3ab4a51a0c041ac2b57bee0"
format_stamp: "Formatted at 2026-09-25 01:18:54 on dist-test-slave-s12s"
server_key: "97f43ea85b2acdaa5fefe6c9db8a4900"
server_key_iv: "f579151df50042121898a3ed26d178bd"
server_key_version: "kuduclusterkey@0"
I20260925 01:18:54.328954 32168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:54.363690 32168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:54.368644 32168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:54.369040 32168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:54.388789 32168 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:44231
I20260925 01:18:54.388816 32234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:44231 every 8 connection(s)
I20260925 01:18:54.390447 32168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:18:54.394580 32235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:18:54.398156 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 32168
I20260925 01:18:54.398366 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:18:54.399220 20890 external_mini_cluster.cc:1428] Setting key bdde14827100e78075c5cce3f1a0632a
I20260925 01:18:54.410629 32235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0: Bootstrap starting.
I20260925 01:18:54.416023 32235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0: Neither blocks nor log segments found. Creating new log.
I20260925 01:18:54.417438 32235 log.cc:824] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0: Log is configured to *not* fsync() on all Append() calls
I20260925 01:18:54.420626 32235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0: No bootstrap required, opened a new log
I20260925 01:18:54.429858 32235 raft_consensus.cc:348] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dac9f4f3ab4a51a0c041ac2b57bee0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44231 } }
I20260925 01:18:54.430300 32235 raft_consensus.cc:374] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:18:54.430440 32235 raft_consensus.cc:729] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7dac9f4f3ab4a51a0c041ac2b57bee0, State: Initialized, Role: FOLLOWER
I20260925 01:18:54.431195 32235 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dac9f4f3ab4a51a0c041ac2b57bee0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44231 } }
I20260925 01:18:54.431490 32235 raft_consensus.cc:388] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Sep 25 01:18:54 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:18:54.431696 32235 raft_consensus.cc:482] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:18:54.431886 32235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:18:54.435353 32235 raft_consensus.cc:504] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dac9f4f3ab4a51a0c041ac2b57bee0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44231 } }
I20260925 01:18:54.436074 32235 leader_election.cc:302] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d7dac9f4f3ab4a51a0c041ac2b57bee0; no voters:
I20260925 01:18:54.436858 32235 leader_election.cc:288] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:18:54.437019 32240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:18:54.445501 32238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:54.401477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.20.102.190:44231)
0925 01:18:54.416723 (+ 15246us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:54.416743 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:54.416798 (+ 55us) server_negotiation.cc:407] Connection header received
0925 01:18:54.419551 (+ 2753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:54.419583 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:54.420111 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:54.420570 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:54.422279 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:54.424523 (+ 2244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:54.426460 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:54.427688 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:54.427914 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:54.432948 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:54.432959 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:54.432982 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:54.433039 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:54.437326 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:54.438627 (+ 1301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:54.438640 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:54.438655 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:54.438904 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:54.439498 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:54.439505 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:54.439512 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:54.440499 (+ 987us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:54.442569 (+ 2070us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:54.442960 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 01:18:54.443584 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14919,"thread_start_us":14647,"threads_started":1}
I20260925 01:18:54.447383 32240 raft_consensus.cc:686] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [term 1 LEADER]: Becoming Leader. State: Replica: d7dac9f4f3ab4a51a0c041ac2b57bee0, State: Running, Role: LEADER
I20260925 01:18:54.448001 32235 sys_catalog.cc:564] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:18:54.448141 32240 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dac9f4f3ab4a51a0c041ac2b57bee0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44231 } }
I20260925 01:18:54.451797 32242 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7dac9f4f3ab4a51a0c041ac2b57bee0. Latest consensus state: current_term: 1 leader_uuid: "d7dac9f4f3ab4a51a0c041ac2b57bee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dac9f4f3ab4a51a0c041ac2b57bee0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44231 } } }
I20260925 01:18:54.452244 32242 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:18:54.452805 32241 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d7dac9f4f3ab4a51a0c041ac2b57bee0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d7dac9f4f3ab4a51a0c041ac2b57bee0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44231 } } }
I20260925 01:18:54.453526 32241 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:18:54.461156 32245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:18:54.467559 32245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:18:54.488440 32245 catalog_manager.cc:1347] Generated new cluster ID: 4083d1a33e0441699de8f0a27417ed92
I20260925 01:18:54.488567 32245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:18:54.508865 32235 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 01:18:54.513486 32245 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:18:54.515384 32245 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:18:54.532148 32245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0: Generated new TSK 0
I20260925 01:18:54.533560 32245 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:18:55.460104 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.102.148:49467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:44231
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:18:55 dist-test-slave-s12s krb5kdc[31724](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299135, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 01:18:55.877300 32280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:55.877746 32280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:55.877928 32280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:55.889561 32280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:55.889783 32280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:55.890045 32280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:18:55.905918 32280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.102.148:49467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:44231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:55.907904 32280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:55.909735 32280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:55.928001 32291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:55.929064 32290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:55.934418 32280 server_base.cc:1034] running on GCE node
W20260925 01:18:55.934893 32293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:55.935940 32280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:55.937654 32280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:55.938966 32280 hybrid_clock.cc:648] HybridClock initialized: now 1790299135938845 us; error 99 us; skew 500 ppm
I20260925 01:18:55.939396 32280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:18:55 dist-test-slave-s12s krb5kdc[31724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299135, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:18:55.943871 32280 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:18:55.953451 32280 webserver.cc:460] Webserver started at http://127.20.102.129:37541/ using document root <none> and password file <none>
I20260925 01:18:55.954414 32280 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:55.954564 32280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:55.954943 32280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:18:55 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299135, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:18:55.984254 32280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "7f6d328fe1204056b5230d2fabc0f30a"
format_stamp: "Formatted at 2026-09-25 01:18:55 on dist-test-slave-s12s"
server_key: "d6a21c77c5c4d7f4f3fe31e5175ea830"
server_key_iv: "ea7fd16f4737be03097ace794671a1ed"
server_key_version: "kuduclusterkey@0"
I20260925 01:18:55.985443 32280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "7f6d328fe1204056b5230d2fabc0f30a"
format_stamp: "Formatted at 2026-09-25 01:18:55 on dist-test-slave-s12s"
server_key: "d6a21c77c5c4d7f4f3fe31e5175ea830"
server_key_iv: "ea7fd16f4737be03097ace794671a1ed"
server_key_version: "kuduclusterkey@0"
I20260925 01:18:55.998929 32280 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.007s
I20260925 01:18:56.035111 32300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:56.037541 32280 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.013s sys 0.004s
I20260925 01:18:56.037775 32280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "7f6d328fe1204056b5230d2fabc0f30a"
format_stamp: "Formatted at 2026-09-25 01:18:55 on dist-test-slave-s12s"
server_key: "d6a21c77c5c4d7f4f3fe31e5175ea830"
server_key_iv: "ea7fd16f4737be03097ace794671a1ed"
server_key_version: "kuduclusterkey@0"
I20260925 01:18:56.038096 32280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:56.094720 32280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:56.103314 32280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:56.103765 32280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:56.105212 32280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:18:56.107241 32280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:18:56.107348 32280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:56.107494 32280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:18:56.107565 32280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:56.216643 32280 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:42019
I20260925 01:18:56.216735 32412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:42019 every 8 connection(s)
I20260925 01:18:56.218532 32280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:18:56.225323 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 32280
I20260925 01:18:56.225549 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:18:56.226532 20890 external_mini_cluster.cc:1428] Setting key fc88365defeefdded9d41bcf3d74821a
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
Sep 25 01:18:56 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299135, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:18:56.276750 32415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:56.229102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:58771 (local address 127.20.102.190:44231)
0925 01:18:56.241177 (+ 12075us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:56.241192 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:56.241218 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:18:56.243926 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:56.243933 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:56.244100 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:56.244218 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:56.254681 (+ 10463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:56.256139 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:56.262740 (+ 6601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:56.263506 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:56.263630 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:56.268667 (+ 5037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:56.268695 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:56.268703 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:56.268754 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:56.272670 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:56.273715 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:56.273723 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:56.273730 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:56.273896 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:56.274432 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:56.274436 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:56.274439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:56.274824 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:56.274962 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:56.276063 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0925 01:18:56.276313 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11920,"thread_start_us":11723,"threads_started":1}
I20260925 01:18:56.279075 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:56.229860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:44231 (local address 127.20.102.129:58771)
0925 01:18:56.240896 (+ 11036us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:56.240962 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:56.243571 (+ 2609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:56.245026 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:56.245081 (+ 55us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:56.245798 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:18:56.250546 (+ 4748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:56.250581 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:56.256485 (+ 5904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:56.256493 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:56.259030 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:56.259059 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:56.264036 (+ 4977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:56.264044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:56.264379 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:56.265315 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:18:56.265349 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:18:56.268368 (+ 3019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:18:56.272893 (+ 4525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:18:56.272905 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:18:56.272926 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:18:56.273570 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:18:56.274044 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:18:56.274049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:18:56.274052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:18:56.274298 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:18:56.275067 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:18:56.275076 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:18:56.275602 (+ 526us) client_negotiation.cc:769] Sending connection context
0925 01:18:56.276980 (+ 1378us) client_negotiation.cc:240] Negotiation successful
0925 01:18:56.277563 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10662,"thread_start_us":10389,"threads_started":1}
I20260925 01:18:56.283645 32413 heartbeater.cc:347] Connected to a master server at 127.20.102.190:44231
I20260925 01:18:56.284044 32413 heartbeater.cc:464] Registering TS with master...
I20260925 01:18:56.285094 32413 heartbeater.cc:511] Master 127.20.102.190:44231 requested a full tablet report, sending...
I20260925 01:18:56.288499 32200 ts_manager.cc:194] Registered new tserver with Master: 7f6d328fe1204056b5230d2fabc0f30a (127.20.102.129:42019)
I20260925 01:18:56.290601 32200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:58771
Sep 25 01:18:56 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299135, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:18:56.324138 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.102.148:49467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:44231
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:18:56 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299135, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
W20260925 01:18:56.833564 32419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:56.833992 32419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:56.834174 32419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:56.843122 32419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:56.843271 32419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:56.843477 32419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:18:56.859249 32419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.102.148:49467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:44231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:56.861127 32419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:56.863113 32419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:56.881306 32430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:56.881696 32428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:56.882694 32436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:56.882917 32419 server_base.cc:1034] running on GCE node
I20260925 01:18:56.883628 32419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:56.884713 32419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:56.885975 32419 hybrid_clock.cc:648] HybridClock initialized: now 1790299136885894 us; error 57 us; skew 500 ppm
I20260925 01:18:56.886320 32419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:18:56 dist-test-slave-s12s krb5kdc[31724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299136, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:18:56.889683 32419 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:18:56.891259 32419 webserver.cc:460] Webserver started at http://127.20.102.130:40929/ using document root <none> and password file <none>
I20260925 01:18:56.891971 32419 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:56.892122 32419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:56.892464 32419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:18:56 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299136, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:18:56.907768 32419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "a7c5afd2f4b142998165aa966b62655f"
format_stamp: "Formatted at 2026-09-25 01:18:56 on dist-test-slave-s12s"
server_key: "53cac271943313d4443c54cdf21a26fb"
server_key_iv: "57808b24403e945910f83f58497d2404"
server_key_version: "kuduclusterkey@0"
I20260925 01:18:56.908789 32419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "a7c5afd2f4b142998165aa966b62655f"
format_stamp: "Formatted at 2026-09-25 01:18:56 on dist-test-slave-s12s"
server_key: "53cac271943313d4443c54cdf21a26fb"
server_key_iv: "57808b24403e945910f83f58497d2404"
server_key_version: "kuduclusterkey@0"
I20260925 01:18:56.918449 32419 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.008s
I20260925 01:18:56.933236 32443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:56.934800 32419 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260925 01:18:56.934989 32419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "a7c5afd2f4b142998165aa966b62655f"
format_stamp: "Formatted at 2026-09-25 01:18:56 on dist-test-slave-s12s"
server_key: "53cac271943313d4443c54cdf21a26fb"
server_key_iv: "57808b24403e945910f83f58497d2404"
server_key_version: "kuduclusterkey@0"
I20260925 01:18:56.935216 32419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:56.954905 32419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:56.959079 32419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:56.959385 32419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:56.960328 32419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:18:56.961823 32419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:18:56.961932 32419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:56.962078 32419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:18:56.962146 32419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:57.005802 32419 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:39579
I20260925 01:18:57.005821 32555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:39579 every 8 connection(s)
I20260925 01:18:57.007160 32419 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:18:57.008652 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 32419
I20260925 01:18:57.008864 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:18:57.009438 20890 external_mini_cluster.cc:1428] Setting key 79e0e85bbe1939fe6e167ee7d8300cd1
Sep 25 01:18:57 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299136, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
I20260925 01:18:57.035151 32558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.012617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:49743 (local address 127.20.102.190:44231)
0925 01:18:57.020966 (+ 8349us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:57.020970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:57.020999 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:18:57.021045 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:57.021048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:57.021104 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:57.021224 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:57.022798 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.023541 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.025138 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.025540 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.025664 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.029238 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:57.029243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:57.029246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:57.029282 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:57.032023 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:57.032808 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:57.032812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:57.032815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:57.032926 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:57.033328 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:57.033332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:57.033335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:57.033693 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:57.033832 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:57.034477 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 01:18:57.034759 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8242,"thread_start_us":8110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
I20260925 01:18:57.036096 32559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.013354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:44231 (local address 127.20.102.130:49743)
0925 01:18:57.016635 (+ 3281us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:57.016723 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:57.020130 (+ 3407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:57.021490 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:57.021504 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:57.022039 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:18:57.022628 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:57.022646 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.023684 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.023693 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:57.024999 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:57.025012 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.025763 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.025772 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:57.026029 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.026780 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:18:57.026803 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:18:57.029022 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:18:57.032157 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:18:57.032165 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:18:57.032181 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:18:57.032663 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:18:57.033032 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:18:57.033037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:18:57.033040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:18:57.033215 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:18:57.033821 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:18:57.033826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:18:57.034123 (+ 297us) client_negotiation.cc:769] Sending connection context
0925 01:18:57.034371 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 01:18:57.034797 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2855,"thread_start_us":2581,"threads_started":1}
I20260925 01:18:57.037626 32556 heartbeater.cc:347] Connected to a master server at 127.20.102.190:44231
I20260925 01:18:57.037961 32556 heartbeater.cc:464] Registering TS with master...
I20260925 01:18:57.038916 32556 heartbeater.cc:511] Master 127.20.102.190:44231 requested a full tablet report, sending...
I20260925 01:18:57.040508 32200 ts_manager.cc:194] Registered new tserver with Master: a7c5afd2f4b142998165aa966b62655f (127.20.102.130:39579)
I20260925 01:18:57.041278 32200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:49743
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:18:57.063868 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.102.148:49467/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:44231
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:36617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:18:57.294548 32413 heartbeater.cc:503] Master 127.20.102.190:44231 was elected leader, sending a full tablet report...
W20260925 01:18:57.406728 32562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:18:57.407012 32562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:18:57.407205 32562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:18:57.414752 32562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:18:57.414865 32562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:18:57.415081 32562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:18:57.425544 32562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:36617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.102.148:49467/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:44231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:18:57.426858 32562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:18:57.428221 32562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:18:57.442128 32567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:57.442293 32570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:18:57.442380 32568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:18:57.444024 32562 server_base.cc:1034] running on GCE node
I20260925 01:18:57.444653 32562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:18:57.445626 32562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:18:57.446851 32562 hybrid_clock.cc:648] HybridClock initialized: now 1790299137446773 us; error 38 us; skew 500 ppm
I20260925 01:18:57.447189 32562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:18:57 dist-test-slave-s12s krb5kdc[31724](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299137, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:18:57.450423 32562 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:18:57.451820 32562 webserver.cc:460] Webserver started at http://127.20.102.131:33701/ using document root <none> and password file <none>
I20260925 01:18:57.452415 32562 fs_manager.cc:359] Metadata directory not provided
I20260925 01:18:57.452515 32562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:18:57.452841 32562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:18:57 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299137, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:18:57.470304 32562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "928fdbbd83114f05abebd6ead8999d48"
format_stamp: "Formatted at 2026-09-25 01:18:57 on dist-test-slave-s12s"
server_key: "4bd82575221851fe52bf43bd17b2506f"
server_key_iv: "c106707f849e8d0670ba8a5d86fa3e2a"
server_key_version: "kuduclusterkey@0"
I20260925 01:18:57.471338 32562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "928fdbbd83114f05abebd6ead8999d48"
format_stamp: "Formatted at 2026-09-25 01:18:57 on dist-test-slave-s12s"
server_key: "4bd82575221851fe52bf43bd17b2506f"
server_key_iv: "c106707f849e8d0670ba8a5d86fa3e2a"
server_key_version: "kuduclusterkey@0"
I20260925 01:18:57.478461 32562 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20260925 01:18:57.493652 32577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:57.495282 32562 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:18:57.495463 32562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "928fdbbd83114f05abebd6ead8999d48"
format_stamp: "Formatted at 2026-09-25 01:18:57 on dist-test-slave-s12s"
server_key: "4bd82575221851fe52bf43bd17b2506f"
server_key_iv: "c106707f849e8d0670ba8a5d86fa3e2a"
server_key_version: "kuduclusterkey@0"
I20260925 01:18:57.495719 32562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:18:57.514040 32562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:18:57.518208 32562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:18:57.518520 32562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:18:57.519518 32562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:18:57.520957 32562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:18:57.521054 32562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:57.521203 32562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:18:57.521272 32562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:18:57.556175 32562 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:36913
I20260925 01:18:57.556226 32689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:36913 every 8 connection(s)
I20260925 01:18:57.557572 32562 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:18:57.565964 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 32562
I20260925 01:18:57.566129 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:18:57.566563 20890 external_mini_cluster.cc:1428] Setting key 61f20f5f08327bd4789569973d987a45
Sep 25 01:18:57 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299137, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:18:57.577050 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.560174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:38245 (local address 127.20.102.190:44231)
0925 01:18:57.560673 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:57.560694 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:57.561497 (+ 803us) server_negotiation.cc:407] Connection header received
0925 01:18:57.563476 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:57.563483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:57.563561 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:57.563677 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:57.565126 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.566136 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.567832 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.568287 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.568461 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.571664 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:57.571668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:57.571671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:57.571703 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:57.574189 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:57.574896 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:57.574900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:57.574902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:57.575023 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:57.575430 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:57.575435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:57.575438 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:57.575687 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:57.575830 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:57.576458 (+ 628us) server_negotiation.cc:299] Negotiation successful
0925 01:18:57.576704 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":281,"threads_started":1}
I20260925 01:18:57.577872 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.560623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:44231 (local address 127.20.102.131:38245)
0925 01:18:57.561345 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:57.561386 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:57.563163 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:57.563908 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:57.563916 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:57.564325 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:18:57.564964 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:57.564982 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.566298 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.566302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:57.567688 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:57.567707 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.568552 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.568561 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:57.568850 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.569540 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:18:57.569561 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:18:57.571484 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:18:57.574349 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:18:57.574357 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:18:57.574370 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:18:57.574770 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:18:57.575127 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:18:57.575132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:18:57.575136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:18:57.575316 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:18:57.575804 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:18:57.575812 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:18:57.576101 (+ 289us) client_negotiation.cc:769] Sending connection context
0925 01:18:57.576329 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 01:18:57.576835 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":278,"threads_started":1}
I20260925 01:18:57.579207 32690 heartbeater.cc:347] Connected to a master server at 127.20.102.190:44231
I20260925 01:18:57.579517 32690 heartbeater.cc:464] Registering TS with master...
I20260925 01:18:57.580230 32690 heartbeater.cc:511] Master 127.20.102.190:44231 requested a full tablet report, sending...
I20260925 01:18:57.581923 32199 ts_manager.cc:194] Registered new tserver with Master: 928fdbbd83114f05abebd6ead8999d48 (127.20.102.131:36913)
I20260925 01:18:57.582834 32199 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:38245
I20260925 01:18:57.594069 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:18:57.611107 32692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.598620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51494 (local address 127.20.102.190:44231)
0925 01:18:57.598782 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:57.598787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:57.599363 (+ 576us) server_negotiation.cc:407] Connection header received
0925 01:18:57.599502 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:57.599507 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:57.599572 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:57.599691 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:57.601234 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.601967 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.603281 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.603675 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.603763 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.606233 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:57.606238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:57.606241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:57.606272 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:57.608762 (+ 2490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:57.609377 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:57.609394 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:57.609397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:57.609511 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:57.609894 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:57.609898 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:57.609901 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:57.610189 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:57.610365 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:57.610694 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 01:18:57.610862 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:18:57.615461 32199 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51494:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:18:57.622617 32199 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:18:57.657414 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.637829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38292 (local address 127.20.102.131:36913)
0925 01:18:57.649661 (+ 11832us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:57.649668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:57.649701 (+ 33us) server_negotiation.cc:407] Connection header received
0925 01:18:57.649798 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:57.649806 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:57.650097 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:57.650295 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:18:57.651155 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.652487 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.654852 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.656300 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.656503 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.656672 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:57.656845 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 01:18:57.657011 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11646,"thread_start_us":11485,"threads_started":1}
I20260925 01:18:57.657469 32707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.637551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:36913 (local address 127.0.0.1:38292)
0925 01:18:57.649148 (+ 11597us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:57.649166 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:57.649274 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:57.650407 (+ 1133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:57.650413 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:57.650440 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:18:57.650915 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:57.650929 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.652622 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.652625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:57.654652 (+ 2027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:57.654665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.656478 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.656482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:57.656610 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.656642 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:18:57.656807 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 01:18:57.657005 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11436,"mutex_wait_us":78,"thread_start_us":11303,"threads_started":1}
I20260925 01:18:57.658157 32704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.635306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:39579 (local address 127.0.0.1:46368)
0925 01:18:57.649305 (+ 13999us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:57.649322 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:57.649440 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:57.650055 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:57.650059 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:57.650094 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:18:57.650611 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:57.650628 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.652704 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.652710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:57.655398 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:57.655411 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.657438 (+ 2027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.657447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:57.657650 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.657678 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 01:18:57.657750 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 01:18:57.657867 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12185,"mutex_wait_us":186,"thread_start_us":12215,"threads_started":1}
I20260925 01:18:57.659267 32705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.635555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.20.102.130:39579)
0925 01:18:57.649264 (+ 13709us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:57.649273 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:57.649595 (+ 322us) server_negotiation.cc:407] Connection header received
0925 01:18:57.649688 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:57.649694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:57.649887 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:57.650062 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:18:57.650841 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.652395 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.655581 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.657010 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.657183 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.658680 (+ 1497us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:57.658752 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 01:18:57.658926 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12484,"thread_start_us":12676,"threads_started":1}
I20260925 01:18:57.661437 32625 tablet_service.cc:1467] Processing CreateTablet for tablet 49fc46008738446cbf78c257b085721c (DEFAULT_TABLE table=test-table [id=546e502f7da6440aafdcac81291c5fa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:18:57.662108 32491 tablet_service.cc:1467] Processing CreateTablet for tablet 49fc46008738446cbf78c257b085721c (DEFAULT_TABLE table=test-table [id=546e502f7da6440aafdcac81291c5fa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:18:57.662842 32625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49fc46008738446cbf78c257b085721c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:18:57.663327 32491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49fc46008738446cbf78c257b085721c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:18:57.669490 32703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.635590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:42019 (local address 127.0.0.1:36782)
0925 01:18:57.647809 (+ 12219us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:57.647875 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:57.648093 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:57.649626 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:57.649633 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:57.649666 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:18:57.650159 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:57.650173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.652523 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.652531 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:57.655397 (+ 2866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:57.655411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.668791 (+ 13380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.668796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:57.668923 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.668947 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:18:57.669031 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 01:18:57.669125 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12078,"mutex_wait_us":176,"thread_start_us":11735,"threads_started":1}
I20260925 01:18:57.669667 32706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.635955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36782 (local address 127.20.102.129:42019)
0925 01:18:57.646861 (+ 10906us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:57.646873 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:57.648293 (+ 1420us) server_negotiation.cc:407] Connection header received
0925 01:18:57.648401 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:57.648409 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:57.648761 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:57.648979 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:18:57.650344 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.652385 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.656799 (+ 4414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.658318 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.658519 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.658698 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:57.669122 (+ 10424us) server_negotiation.cc:299] Negotiation successful
0925 01:18:57.669332 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10662,"thread_start_us":10920,"threads_started":1}
I20260925 01:18:57.673887 32348 tablet_service.cc:1467] Processing CreateTablet for tablet 49fc46008738446cbf78c257b085721c (DEFAULT_TABLE table=test-table [id=546e502f7da6440aafdcac81291c5fa9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:18:57.674574 32709 tablet_bootstrap.cc:492] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f: Bootstrap starting.
I20260925 01:18:57.675522 32348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49fc46008738446cbf78c257b085721c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:18:57.676084 32710 tablet_bootstrap.cc:492] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48: Bootstrap starting.
I20260925 01:18:57.678933 32709 tablet_bootstrap.cc:654] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f: Neither blocks nor log segments found. Creating new log.
I20260925 01:18:57.679253 32710 tablet_bootstrap.cc:654] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48: Neither blocks nor log segments found. Creating new log.
I20260925 01:18:57.680186 32710 log.cc:824] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48: Log is configured to *not* fsync() on all Append() calls
I20260925 01:18:57.680172 32709 log.cc:824] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:18:57.683113 32709 tablet_bootstrap.cc:492] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f: No bootstrap required, opened a new log
I20260925 01:18:57.683346 32710 tablet_bootstrap.cc:492] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48: No bootstrap required, opened a new log
I20260925 01:18:57.683396 32709 ts_tablet_manager.cc:1397] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 01:18:57.683624 32710 ts_tablet_manager.cc:1397] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260925 01:18:57.691110 32713 tablet_bootstrap.cc:492] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a: Bootstrap starting.
I20260925 01:18:57.690843 32709 raft_consensus.cc:348] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c5afd2f4b142998165aa966b62655f" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 39579 } } peers { permanent_uuid: "7f6d328fe1204056b5230d2fabc0f30a" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 42019 } } peers { permanent_uuid: "928fdbbd83114f05abebd6ead8999d48" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36913 } }
I20260925 01:18:57.691289 32709 raft_consensus.cc:374] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:18:57.691432 32709 raft_consensus.cc:729] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c5afd2f4b142998165aa966b62655f, State: Initialized, Role: FOLLOWER
I20260925 01:18:57.692240 32709 consensus_queue.cc:260] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c5afd2f4b142998165aa966b62655f" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 39579 } } peers { permanent_uuid: "7f6d328fe1204056b5230d2fabc0f30a" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 42019 } } peers { permanent_uuid: "928fdbbd83114f05abebd6ead8999d48" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36913 } }
I20260925 01:18:57.693415 32556 heartbeater.cc:503] Master 127.20.102.190:44231 was elected leader, sending a full tablet report...
I20260925 01:18:57.693826 32710 raft_consensus.cc:348] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c5afd2f4b142998165aa966b62655f" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 39579 } } peers { permanent_uuid: "7f6d328fe1204056b5230d2fabc0f30a" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 42019 } } peers { permanent_uuid: "928fdbbd83114f05abebd6ead8999d48" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36913 } }
I20260925 01:18:57.694120 32709 ts_tablet_manager.cc:1428] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f: Time spent starting tablet: real 0.011s user 0.001s sys 0.010s
I20260925 01:18:57.694222 32710 raft_consensus.cc:374] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:18:57.694326 32710 raft_consensus.cc:729] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 928fdbbd83114f05abebd6ead8999d48, State: Initialized, Role: FOLLOWER
I20260925 01:18:57.694895 32710 consensus_queue.cc:260] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c5afd2f4b142998165aa966b62655f" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 39579 } } peers { permanent_uuid: "7f6d328fe1204056b5230d2fabc0f30a" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 42019 } } peers { permanent_uuid: "928fdbbd83114f05abebd6ead8999d48" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36913 } }
I20260925 01:18:57.695014 32713 tablet_bootstrap.cc:654] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a: Neither blocks nor log segments found. Creating new log.
I20260925 01:18:57.695979 32690 heartbeater.cc:503] Master 127.20.102.190:44231 was elected leader, sending a full tablet report...
I20260925 01:18:57.696587 32710 ts_tablet_manager.cc:1428] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260925 01:18:57.697106 32713 log.cc:824] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a: Log is configured to *not* fsync() on all Append() calls
I20260925 01:18:57.700806 32713 tablet_bootstrap.cc:492] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a: No bootstrap required, opened a new log
I20260925 01:18:57.701026 32713 ts_tablet_manager.cc:1397] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260925 01:18:57.704334 32715 raft_consensus.cc:482] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:18:57.704560 32715 raft_consensus.cc:504] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c5afd2f4b142998165aa966b62655f" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 39579 } } peers { permanent_uuid: "7f6d328fe1204056b5230d2fabc0f30a" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 42019 } } peers { permanent_uuid: "928fdbbd83114f05abebd6ead8999d48" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36913 } }
I20260925 01:18:57.706059 32715 leader_election.cc:288] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7c5afd2f4b142998165aa966b62655f (127.20.102.130:39579), 7f6d328fe1204056b5230d2fabc0f30a (127.20.102.129:42019)
I20260925 01:18:57.709120 32713 raft_consensus.cc:348] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c5afd2f4b142998165aa966b62655f" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 39579 } } peers { permanent_uuid: "7f6d328fe1204056b5230d2fabc0f30a" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 42019 } } peers { permanent_uuid: "928fdbbd83114f05abebd6ead8999d48" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36913 } }
I20260925 01:18:57.709525 32713 raft_consensus.cc:374] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:18:57.709635 32713 raft_consensus.cc:729] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6d328fe1204056b5230d2fabc0f30a, State: Initialized, Role: FOLLOWER
I20260925 01:18:57.710417 32713 consensus_queue.cc:260] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c5afd2f4b142998165aa966b62655f" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 39579 } } peers { permanent_uuid: "7f6d328fe1204056b5230d2fabc0f30a" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 42019 } } peers { permanent_uuid: "928fdbbd83114f05abebd6ead8999d48" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36913 } }
I20260925 01:18:57.712646 32713 ts_tablet_manager.cc:1428] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20260925 01:18:57.713625 32706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.706600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:59651 (local address 127.20.102.129:42019)
0925 01:18:57.706791 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:57.706799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:57.706819 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:18:57.706899 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:57.706905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:57.706989 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:57.707091 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:18:57.707629 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.708786 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.711039 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.712490 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.712668 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.712784 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:57.713272 (+ 488us) server_negotiation.cc:299] Negotiation successful
0925 01:18:57.713403 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:18:57.713622 32693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.706309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:42019 (local address 127.20.102.131:59651)
0925 01:18:57.706547 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:57.706583 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:57.706698 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:57.707240 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:57.707251 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:57.707290 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:18:57.707485 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:57.707495 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.708911 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.708916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:57.710824 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:57.710840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.712889 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.712895 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:57.713078 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.713115 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 01:18:57.713204 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 01:18:57.713338 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":48}
I20260925 01:18:57.714625 32705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.706919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:44335 (local address 127.20.102.130:39579)
0925 01:18:57.707128 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:57.707135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:57.707494 (+ 359us) server_negotiation.cc:407] Connection header received
0925 01:18:57.707733 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:57.707740 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:57.707818 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:57.707930 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:18:57.708498 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.709569 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.711964 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.713468 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.713590 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.713659 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:57.714259 (+ 600us) server_negotiation.cc:299] Negotiation successful
0925 01:18:57.714398 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:18:57.714764 32368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49fc46008738446cbf78c257b085721c" candidate_uuid: "928fdbbd83114f05abebd6ead8999d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6d328fe1204056b5230d2fabc0f30a" is_pre_election: true
I20260925 01:18:57.715297 32368 raft_consensus.cc:2454] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 928fdbbd83114f05abebd6ead8999d48 in term 0.
I20260925 01:18:57.716007 32717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.706390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:39579 (local address 127.20.102.131:44335)
0925 01:18:57.707402 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0925 01:18:57.707415 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 01:18:57.707544 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:18:57.707963 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:18:57.707970 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:18:57.708000 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:18:57.708336 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:57.708347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.709734 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.709740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:57.711784 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:18:57.711798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.713850 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.713857 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:18:57.714039 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.714078 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 01:18:57.714222 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 01:18:57.715662 (+ 1440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":29,"thread_start_us":592,"threads_started":1}
I20260925 01:18:57.715978 32580 leader_election.cc:302] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6d328fe1204056b5230d2fabc0f30a, 928fdbbd83114f05abebd6ead8999d48; no voters:
I20260925 01:18:57.716444 32715 raft_consensus.cc:2781] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:18:57.716599 32715 raft_consensus.cc:482] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:18:57.716738 32715 raft_consensus.cc:3037] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:18:57.717058 32511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49fc46008738446cbf78c257b085721c" candidate_uuid: "928fdbbd83114f05abebd6ead8999d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c5afd2f4b142998165aa966b62655f" is_pre_election: true
I20260925 01:18:57.717558 32511 raft_consensus.cc:2454] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 928fdbbd83114f05abebd6ead8999d48 in term 0.
I20260925 01:18:57.719292 32715 raft_consensus.cc:504] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c5afd2f4b142998165aa966b62655f" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 39579 } } peers { permanent_uuid: "7f6d328fe1204056b5230d2fabc0f30a" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 42019 } } peers { permanent_uuid: "928fdbbd83114f05abebd6ead8999d48" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36913 } }
I20260925 01:18:57.719837 32715 leader_election.cc:288] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [CANDIDATE]: Term 1 election: Requested vote from peers a7c5afd2f4b142998165aa966b62655f (127.20.102.130:39579), 7f6d328fe1204056b5230d2fabc0f30a (127.20.102.129:42019)
I20260925 01:18:57.720269 32368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49fc46008738446cbf78c257b085721c" candidate_uuid: "928fdbbd83114f05abebd6ead8999d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6d328fe1204056b5230d2fabc0f30a"
I20260925 01:18:57.720266 32511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49fc46008738446cbf78c257b085721c" candidate_uuid: "928fdbbd83114f05abebd6ead8999d48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7c5afd2f4b142998165aa966b62655f"
I20260925 01:18:57.720463 32368 raft_consensus.cc:3037] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:18:57.720464 32511 raft_consensus.cc:3037] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:18:57.723687 32368 raft_consensus.cc:2454] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 928fdbbd83114f05abebd6ead8999d48 in term 1.
I20260925 01:18:57.723688 32511 raft_consensus.cc:2454] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 928fdbbd83114f05abebd6ead8999d48 in term 1.
I20260925 01:18:57.724090 32581 leader_election.cc:302] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 928fdbbd83114f05abebd6ead8999d48, a7c5afd2f4b142998165aa966b62655f; no voters:
I20260925 01:18:57.724408 32715 raft_consensus.cc:2781] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:18:57.724884 32715 raft_consensus.cc:686] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [term 1 LEADER]: Becoming Leader. State: Replica: 928fdbbd83114f05abebd6ead8999d48, State: Running, Role: LEADER
I20260925 01:18:57.725320 32715 consensus_queue.cc:237] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c5afd2f4b142998165aa966b62655f" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 39579 } } peers { permanent_uuid: "7f6d328fe1204056b5230d2fabc0f30a" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 42019 } } peers { permanent_uuid: "928fdbbd83114f05abebd6ead8999d48" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36913 } }
W20260925 01:18:57.727883 32414 tablet.cc:2369] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:18:57.729931 32199 catalog_manager.cc:5495] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 928fdbbd83114f05abebd6ead8999d48 (127.20.102.131). New cstate: current_term: 1 leader_uuid: "928fdbbd83114f05abebd6ead8999d48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7c5afd2f4b142998165aa966b62655f" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 39579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f6d328fe1204056b5230d2fabc0f30a" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 42019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "928fdbbd83114f05abebd6ead8999d48" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 36913 } health_report { overall_health: HEALTHY } } }
I20260925 01:18:57.759084 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.753633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.20.102.131:36913)
0925 01:18:57.753826 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:57.753830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:57.753848 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:18:57.753973 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:57.753977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:57.754068 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:57.754184 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:18:57.754606 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.755325 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.756829 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.757256 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.757372 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.757935 (+ 563us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:18:57.758508 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:18:57.758633 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:57.758715 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 01:18:57.758788 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 01:18:57.759248 32557 tablet.cc:2369] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:18:57.767789 32368 raft_consensus.cc:1261] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a [term 1 FOLLOWER]: Refusing update from remote peer 928fdbbd83114f05abebd6ead8999d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:18:57.767789 32511 raft_consensus.cc:1261] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f [term 1 FOLLOWER]: Refusing update from remote peer 928fdbbd83114f05abebd6ead8999d48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:18:57.768858 32719 consensus_queue.cc:1035] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6d328fe1204056b5230d2fabc0f30a" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 42019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:18:57.769234 32715 consensus_queue.cc:1035] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7c5afd2f4b142998165aa966b62655f" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 39579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:18:57.789271 32724 mvcc.cc:205] Tried to move back new op lower bound from 7333065268286808064 to 7333065268126609408. Current Snapshot: MvccSnapshot[applied={T|T < 7333065268286808064 or (T in {7333065268286808064})}]
I20260925 01:18:57.795209 32199 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:18:57.795449 32199 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51494:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:18:57.798928 32199 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d7dac9f4f3ab4a51a0c041ac2b57bee0: Sending DeleteTablet for 3 replicas of tablet 49fc46008738446cbf78c257b085721c
I20260925 01:18:57.800009 32491 tablet_service.cc:1514] Processing DeleteTablet for tablet 49fc46008738446cbf78c257b085721c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:18:57 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:46368
I20260925 01:18:57.800184 32348 tablet_service.cc:1514] Processing DeleteTablet for tablet 49fc46008738446cbf78c257b085721c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:18:57 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:36782
I20260925 01:18:57.800465 32625 tablet_service.cc:1514] Processing DeleteTablet for tablet 49fc46008738446cbf78c257b085721c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:18:57 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:38292
I20260925 01:18:57.805768 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 32280
I20260925 01:18:57.812268 32731 tablet_replica.cc:330] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f: stopping tablet replica
I20260925 01:18:57.812830 32731 raft_consensus.cc:2229] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:18:57.813205 32732 tablet_replica.cc:330] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a: stopping tablet replica
I20260925 01:18:57.813547 32731 raft_consensus.cc:2258] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:18:57.813784 32732 raft_consensus.cc:2229] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:18:57.814313 32732 raft_consensus.cc:2258] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:18:57.816028 32731 ts_tablet_manager.cc:1905] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:18:57.816489 32732 ts_tablet_manager.cc:1905] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:18:57.823781 32732 ts_tablet_manager.cc:1918] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:18:57.824043 32732 log.cc:1196] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/49fc46008738446cbf78c257b085721c
I20260925 01:18:57.824760 32732 ts_tablet_manager.cc:1939] T 49fc46008738446cbf78c257b085721c P 7f6d328fe1204056b5230d2fabc0f30a: Deleting consensus metadata
I20260925 01:18:57.827042 32186 catalog_manager.cc:4841] TS 7f6d328fe1204056b5230d2fabc0f30a (127.20.102.129:42019): tablet 49fc46008738446cbf78c257b085721c (table test-table [id=546e502f7da6440aafdcac81291c5fa9]) successfully deleted
I20260925 01:18:57.829305 32731 ts_tablet_manager.cc:1918] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:18:57.829519 32731 log.cc:1196] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/49fc46008738446cbf78c257b085721c
I20260925 01:18:57.830142 32731 ts_tablet_manager.cc:1939] T 49fc46008738446cbf78c257b085721c P a7c5afd2f4b142998165aa966b62655f: Deleting consensus metadata
I20260925 01:18:57.832463 32187 catalog_manager.cc:4841] TS a7c5afd2f4b142998165aa966b62655f (127.20.102.130:39579): tablet 49fc46008738446cbf78c257b085721c (table test-table [id=546e502f7da6440aafdcac81291c5fa9]) successfully deleted
I20260925 01:18:57.834190 32733 tablet_replica.cc:330] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48: stopping tablet replica
I20260925 01:18:57.834728 32733 raft_consensus.cc:2229] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:18:57.835503 32733 raft_consensus.cc:2258] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:18:57.837666 32733 ts_tablet_manager.cc:1905] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 01:18:57 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:18:57.845346 32733 ts_tablet_manager.cc:1918] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:18:57.845541 32733 log.cc:1196] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/49fc46008738446cbf78c257b085721c
I20260925 01:18:57.846163 32733 ts_tablet_manager.cc:1939] T 49fc46008738446cbf78c257b085721c P 928fdbbd83114f05abebd6ead8999d48: Deleting consensus metadata
I20260925 01:18:57.847836 32706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.806731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.20.102.129:42019)
0925 01:18:57.806900 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:57.806906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:57.821270 (+ 14364us) server_negotiation.cc:407] Connection header received
0925 01:18:57.821642 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:57.821650 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:57.821777 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:57.821886 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:57.832760 (+ 10874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.834120 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.835843 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.836409 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.836565 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.841347 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:57.841360 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:57.841374 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:57.841423 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:57.844530 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:57.845278 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:57.845285 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:57.845287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:57.845415 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:57.845911 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:57.845916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:57.845920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:57.846434 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:57.846754 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:57.847065 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 01:18:57.847317 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:18:57.848119 32185 catalog_manager.cc:4841] TS 928fdbbd83114f05abebd6ead8999d48 (127.20.102.131:36913): tablet 49fc46008738446cbf78c257b085721c (table test-table [id=546e502f7da6440aafdcac81291c5fa9]) successfully deleted
I20260925 01:18:57.848528 32408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:57.938232 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 32280
I20260925 01:18:57.956946 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 32419
Sep 25 01:18:57 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:18:57.973259 32705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:57.957865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.20.102.130:39579)
0925 01:18:57.958051 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:57.958055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:57.958138 (+ 83us) server_negotiation.cc:407] Connection header received
0925 01:18:57.958318 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:57.958322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:57.958394 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:57.958478 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:57.960098 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.960901 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.962251 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:57.962628 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:57.962763 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:57.967131 (+ 4368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:57.967146 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:57.967161 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:57.967193 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:57.970305 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:57.970965 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:57.970974 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:57.970977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:57.971099 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:57.971492 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:57.971501 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:57.971504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:57.971898 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:57.972220 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:57.972506 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 01:18:57.972994 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:18:57.974013 32551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:58.062862 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 32419
I20260925 01:18:58.086454 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 32562
Sep 25 01:18:58 dist-test-slave-s12s krb5kdc[31724](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:18:58.105881 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:18:58.087374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.20.102.131:36913)
0925 01:18:58.087533 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 01:18:58.087537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:18:58.087554 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:18:58.087667 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:18:58.087671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:18:58.087747 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:18:58.087832 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:18:58.089731 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:58.090544 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:58.092476 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:18:58.093060 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:18:58.093166 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:18:58.098389 (+ 5223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:18:58.098403 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:18:58.098425 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:18:58.098490 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:18:58.102801 (+ 4311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:58.103546 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:58.103555 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:58.103559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:58.103739 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:18:58.104119 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:18:58.104124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:18:58.104128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:18:58.104630 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:18:58.104986 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0925 01:18:58.105245 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 01:18:58.105471 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 01:18:58.106758 32685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:58.232347 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 32562
I20260925 01:18:58.252872 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 32168
I20260925 01:18:58.254091 32230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:18:58.367636 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 32168
I20260925 01:18:58.392951 20890 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 01:18:58.727761 20890 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 01:18:58.727952 20890 mini_ranger.cc:66] Stopping Ranger...
I20260925 01:18:59.072762 20890 mini_ranger.cc:68] Stopped Ranger
2026-09-25 01:18:59.073 UTC [31754] LOG: received smart shutdown request
2026-09-25 01:18:59.076 UTC [31754] LOG: background worker "logical replication launcher" (PID 31763) exited with exit code 1
2026-09-25 01:18:59.077 UTC [31758] LOG: shutting down
2026-09-25 01:18:59.235 UTC [31754] LOG: database system is shut down
2026-09-25T01:18:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49546 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:18:59 dist-test-slave-s12s krb5kdc[32747](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:18:59 dist-test-slave-s12s krb5kdc[32747](info): set up 2 sockets
Sep 25 01:18:59 dist-test-slave-s12s krb5kdc[32747](info): commencing operation
krb5kdc: starting...
W20260925 01:19:00.465250 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:19:00 dist-test-slave-s12s krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:19:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:19:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.20.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.20.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/rangeradmin_127.20.102.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/rangeradmin_127.20.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.20.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.20.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/rangerlookup_127.20.102.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/rangerlookup_127.20.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.20.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/HTTP_127.20.102.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/HTTP_127.20.102.148@KRBTEST.COM.keytab.
I20260925 01:19:00.785884 20890 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task_TC7zy/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/postgres -l logfile start
2026-09-25 01:19:03.195 UTC [309] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 01:19:03.195 UTC [309] LOG: listening on IPv4 address "127.20.102.148", port 51431
2026-09-25 01:19:03.198 UTC [309] LOG: listening on Unix socket "/tmp/.s.PGSQL.51431"
2026-09-25 01:19:03.220 UTC [312] LOG: database system was shut down at 2026-09-25 01:19:01 UTC
2026-09-25 01:19:03.224 UTC [309] LOG: database system is ready to accept connections
I20260925 01:19:04.196996 20890 mini_postgres.cc:96] Postgres bound to 51431
2026-09-25 01:19:04.219 UTC [320] FATAL: database "slave" does not exist
127.20.102.148:51431 - accepting connections
I20260925 01:19:04.220297 20890 mini_ranger.cc:161] Starting Ranger...
I20260925 01:19:04.247088 20890 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 01:19:04.713272 20890 mini_ranger.cc:87] Created ranger Postgres database
I20260925 01:19:04.713413 20890 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-admin
2026-09-25 01:19:04,994 [I] DB FLAVOR :POSTGRES
2026-09-25 01:19:04,994 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 01:19:04,994 [I] Checking connection..
2026-09-25 01:19:04,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 01:19:05,255 [I] Checking connection passed.
2026-09-25 01:19:05,255 [I] --------- Verifying version history table ---------
2026-09-25 01:19:05,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:19:05,533 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 01:19:05,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:19:05,796 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 01:19:05,796 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 01:19:05,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 01:19:06.073 UTC [384] WARNING: there is no transaction in progress
2026-09-25 01:19:06,080 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 01:19:06,080 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 01:19:06,323 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 01:19:06,323 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 01:19:06,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 01:19:06,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 01:19:06,794 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s12s.c.gcp-upstream.internal','N') ;"
2026-09-25 01:19:07,022 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 01:19:07,286 [I] Table x_portal_user does not exist in database ranger
2026-09-25 01:19:07,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 01:19:07,520 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 01:19:07,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 01:19:07,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 01:19:08,044 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 01:19:08,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 01:19:08.994 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.010 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.026 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.041 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.056 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.072 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.132 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.139 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.153 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.163 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.174 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.188 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.198 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.205 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.212 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09.219 UTC [492] WARNING: there is no transaction in progress
2026-09-25 01:19:09,840 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 01:19:09,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 01:19:10,063 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 01:19:10,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 01:19:10,318 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 01:19:10,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 01:19:10,583 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 01:19:10,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 01:19:10,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 01:19:11,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s12s.c.gcp-upstream.internal';"
2026-09-25 01:19:11,317 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 01:19:11,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 01:19:11,571 [I] DB_PATCHES have already been applied
I20260925 01:19:11.577689 20890 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-admin:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task_TC7zy/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 01:19:11.577778 20890 mini_ranger.cc:193] Using host: 127.20.102.148
I20260925 01:19:11.593685 20890 mini_ranger.cc:227] Ranger admin URL: http://127.20.102.148:43047
Sep 25, 2026 1:19:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:19:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:19:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 1:19:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 1:19:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 1:19:12 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 1:19:12 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 01:19:12.723423 20890 mini_ranger.cc:160] Time spent starting Ranger: real 8.503s user 0.005s sys 0.036s
Sep 25, 2026 1:19:32 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 1:19:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 25, 2026 1:19:32 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 1:19:32 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 1:19:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 1:19:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 1:19:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 1:19:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 1:19:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 1:19:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 1:19:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 1:19:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 1:19:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 1:19:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 1:19:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 1:19:32 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 1:19:32 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 1:19:33 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 01:19:33.910418 20890 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.20.102.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.20.102.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/rangerkms_127.20.102.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.20.102.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/rangerkms_127.20.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.20.102.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.20.102.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.20.102.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/HTTP_127.20.102.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.20.102.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/HTTP_127.20.102.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 01:19:34 dist-test-slave-s12s krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299174, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 01:19:34.079146 20890 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 01:19:34.099251 20890 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 01:19:34.915825 20890 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 01:19:34.915974 20890 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-kms
I20260925 01:19:34.916028 20890 mini_ranger_kms.cc:227] Using postgres at 127.20.102.148:51431
2026-09-25 01:19:35,267 [I] DB FLAVOR :POSTGRES
2026-09-25 01:19:35,267 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 01:19:35,267 [I] Checking connection
2026-09-25 01:19:35,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 01:19:35,498 [I] connection success
2026-09-25 01:19:35,499 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 01:19:35,499 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 01:19:35,499 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 01:19:35,745 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 01:19:35,746 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 01:19:35,746 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 01:19:35,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.20.102.148:51431/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 01:19:36,010 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 01:19:36.478519 20890 mini_ranger_kms.cc:310] Created kms service
I20260925 01:19:36.652922 20890 mini_ranger_kms.cc:326] Created kudu user
I20260925 01:19:36.699556 20890 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 01:19:36 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299174, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:19:37.108119 20890 mini_ranger_kms.cc:384] Added ranger policy
I20260925 01:19:37.108474 20890 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-kms:/tmp/dist-test-task_TC7zy/build/asan/bin/postgresql.jar:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task_TC7zy/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 01:19:37.108570 20890 mini_ranger_kms.cc:240] Using host: 127.20.102.148
I20260925 01:19:37.128289 20890 mini_ranger_kms.cc:276] Ranger KMS PID: 672
I20260925 01:19:37.128437 20890 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.20.102.148:36949
Sep 25, 2026 1:19:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 1:19:38 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 1:19:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 1:19:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 1:19:38 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 1:19:38 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 01:19:39 dist-test-slave-s12s krb5kdc[32747](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299179, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.102.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:39.413452 20890 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.334s user 0.066s sys 0.297s
I20260925 01:19:39.413731 20890 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 01:19:39.413940 20890 mini_ranger_kms.cc:401] 127.20.102.148:36949/kms/v1/keys
Sep 25 01:19:39 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299179, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.102.148@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
Sep 25 01:19:40 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299179, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.20.102.148@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
Sep 25 01:19:42 dist-test-slave-s12s krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:19:42.376174 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:43283
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:43283
--ranger_config_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.102.148:36949/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:19:42.708310 723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:19:42.708603 723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:19:42.708747 723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:19:42.715935 723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:19:42.716037 723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:19:42.716089 723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:19:42.716146 723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:19:42.716193 723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:19:42.726649 723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/wal
--ranger_kms_url=127.20.102.148:36949/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:43283
--ranger_config_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:43283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:19:42.727973 723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:19:42.729364 723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:19:42.741624 728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:42.741590 731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:42.741560 729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:19:42.742624 723 server_base.cc:1034] running on GCE node
I20260925 01:19:42.743439 723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:19:42.744992 723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:19:42.746204 723 hybrid_clock.cc:648] HybridClock initialized: now 1790299182746164 us; error 27 us; skew 500 ppm
I20260925 01:19:42.746574 723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:19:42 dist-test-slave-s12s krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299182, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:42.755307 723 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:19:42.757085 723 webserver.cc:460] Webserver started at http://127.20.102.190:35469/ using document root <none> and password file <none>
I20260925 01:19:42.757778 723 fs_manager.cc:359] Metadata directory not provided
I20260925 01:19:42.757907 723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:19:42.758247 723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:19:42 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299182, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:19:42.816206 723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "5245497731624901ac63a98a6db7f2a0"
format_stamp: "Formatted at 2026-09-25 01:19:42 on dist-test-slave-s12s"
server_key: "1747dbe48ceff461be158e1106491bc4"
server_key_iv: "ef2d0c098a87c7c664571efb6077b033"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:42.817101 723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "5245497731624901ac63a98a6db7f2a0"
format_stamp: "Formatted at 2026-09-25 01:19:42 on dist-test-slave-s12s"
server_key: "1747dbe48ceff461be158e1106491bc4"
server_key_iv: "ef2d0c098a87c7c664571efb6077b033"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:42.822916 723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:19:42.839380 738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:42.841315 723 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 01:19:42.841522 723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "5245497731624901ac63a98a6db7f2a0"
format_stamp: "Formatted at 2026-09-25 01:19:42 on dist-test-slave-s12s"
server_key: "1747dbe48ceff461be158e1106491bc4"
server_key_iv: "ef2d0c098a87c7c664571efb6077b033"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:42.841786 723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:19:42.869328 723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:19:42.874642 723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:19:42.874990 723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:19:42.896104 723 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:43283
I20260925 01:19:42.896099 789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:43283 every 8 connection(s)
I20260925 01:19:42.897979 723 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:19:42.901264 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 723
I20260925 01:19:42.901500 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:19:42.902407 20890 external_mini_cluster.cc:1428] Setting key 3d6df1cea6c5de4b943fa43b2c6331ee
I20260925 01:19:42.902765 790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:19:42.920459 790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0: Bootstrap starting.
I20260925 01:19:42.924358 790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0: Neither blocks nor log segments found. Creating new log.
I20260925 01:19:42.925689 790 log.cc:824] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0: Log is configured to *not* fsync() on all Append() calls
I20260925 01:19:42.929160 790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0: No bootstrap required, opened a new log
Sep 25 01:19:42 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:19:42.939069 790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5245497731624901ac63a98a6db7f2a0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 43283 } }
I20260925 01:19:42.939528 790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:19:42.939647 790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5245497731624901ac63a98a6db7f2a0, State: Initialized, Role: FOLLOWER
I20260925 01:19:42.940378 790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5245497731624901ac63a98a6db7f2a0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 43283 } }
I20260925 01:19:42.940750 790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:19:42.940939 790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:19:42.941202 790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:19:42.942633 793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:42.904606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48720 (local address 127.20.102.190:43283)
0925 01:19:42.917748 (+ 13142us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:42.917766 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:42.917832 (+ 66us) server_negotiation.cc:407] Connection header received
0925 01:19:42.920206 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:42.920238 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:42.920806 (+ 568us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:42.921271 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:42.923102 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:42.925161 (+ 2059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:42.926925 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:42.927906 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:42.928107 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:42.932948 (+ 4841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:42.932958 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:42.932978 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:42.933026 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:42.937340 (+ 4314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:42.938146 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:42.938155 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:42.938163 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:42.938346 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:42.938783 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:42.938787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:42.938790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:42.939558 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:42.939866 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:42.940596 (+ 730us) server_negotiation.cc:299] Negotiation successful
0925 01:19:42.941248 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12839,"thread_start_us":12566,"threads_started":1}
I20260925 01:19:42.944044 790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5245497731624901ac63a98a6db7f2a0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 43283 } }
I20260925 01:19:42.944744 790 leader_election.cc:302] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5245497731624901ac63a98a6db7f2a0; no voters:
I20260925 01:19:42.949895 790 leader_election.cc:288] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:19:42.950002 795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:19:42.950748 795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 1 LEADER]: Becoming Leader. State: Replica: 5245497731624901ac63a98a6db7f2a0, State: Running, Role: LEADER
I20260925 01:19:42.951408 795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5245497731624901ac63a98a6db7f2a0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 43283 } }
I20260925 01:19:42.959307 790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:19:42.963220 796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5245497731624901ac63a98a6db7f2a0. Latest consensus state: current_term: 1 leader_uuid: "5245497731624901ac63a98a6db7f2a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5245497731624901ac63a98a6db7f2a0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 43283 } } }
I20260925 01:19:42.963703 796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:19:42.978137 805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:19:42.978688 797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5245497731624901ac63a98a6db7f2a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5245497731624901ac63a98a6db7f2a0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 43283 } } }
I20260925 01:19:42.979148 797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:19:42.989431 805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:19:43.014765 805 catalog_manager.cc:1347] Generated new cluster ID: d9bf2a24084e48b9a29eca9756a61b8c
I20260925 01:19:43.014946 805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:19:43.029765 790 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 01:19:43.048799 805 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:19:43.051218 805 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:19:43.065330 805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0: Generated new TSK 0
I20260925 01:19:43.066282 805 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:19:43.917467 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.102.148:36949/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:43283
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:19:44 dist-test-slave-s12s krb5kdc[32747](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299184, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 01:19:44.346860 836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:19:44.347275 836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:19:44.347424 836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:19:44.359309 836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:19:44.359469 836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:19:44.359668 836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:19:44.375681 836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.20.102.148:36949/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:19:44.377497 836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:19:44.379179 836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:19:44.400112 846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:44.401199 847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:44.409308 849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:19:44.409931 836 server_base.cc:1034] running on GCE node
I20260925 01:19:44.410794 836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:19:44.413012 836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:19:44.414268 836 hybrid_clock.cc:648] HybridClock initialized: now 1790299184414197 us; error 59 us; skew 500 ppm
I20260925 01:19:44.414665 836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:19:44 dist-test-slave-s12s krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299184, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:44.418954 836 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:19:44.422726 836 webserver.cc:460] Webserver started at http://127.20.102.129:42025/ using document root <none> and password file <none>
I20260925 01:19:44.423733 836 fs_manager.cc:359] Metadata directory not provided
I20260925 01:19:44.423878 836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:19:44.424278 836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:19:44 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299184, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:19:44.445318 836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "55dd0c09359d489abeaddf0f3580fd0b"
format_stamp: "Formatted at 2026-09-25 01:19:44 on dist-test-slave-s12s"
server_key: "b06fe42eac09e9d322ecb69ec749995c"
server_key_iv: "d07aeca46580773e65fe7409fa4747fa"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:44.446484 836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "55dd0c09359d489abeaddf0f3580fd0b"
format_stamp: "Formatted at 2026-09-25 01:19:44 on dist-test-slave-s12s"
server_key: "b06fe42eac09e9d322ecb69ec749995c"
server_key_iv: "d07aeca46580773e65fe7409fa4747fa"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:44.460289 836 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.013s sys 0.000s
I20260925 01:19:44.489614 856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:44.492233 836 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.002s sys 0.004s
I20260925 01:19:44.492455 836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "55dd0c09359d489abeaddf0f3580fd0b"
format_stamp: "Formatted at 2026-09-25 01:19:44 on dist-test-slave-s12s"
server_key: "b06fe42eac09e9d322ecb69ec749995c"
server_key_iv: "d07aeca46580773e65fe7409fa4747fa"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:44.492774 836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:19:44.511070 836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:19:44.516656 836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:19:44.517136 836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:19:44.519565 836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:19:44.521530 836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:19:44.521627 836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:44.521773 836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:19:44.521844 836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:19:44 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299184, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:19:44.664927 836 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:41611
I20260925 01:19:44.664944 968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:41611 every 8 connection(s)
I20260925 01:19:44.666440 836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:19:44.670634 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 836
I20260925 01:19:44.670845 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:19:44.671737 20890 external_mini_cluster.cc:1428] Setting key 9a45ce048623c3f908c69cb4ed63b376
Sep 25 01:19:44 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299184, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:19:44.701634 971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:44.672740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:43283 (local address 127.20.102.129:55053)
0925 01:19:44.676312 (+ 3572us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:44.676367 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:44.678821 (+ 2454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:44.681721 (+ 2900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:44.681735 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:44.682378 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:44.683206 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:44.683230 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:44.685140 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:44.685145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:44.687124 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:44.687139 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:44.688380 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:44.688386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:44.688708 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:44.689513 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:44.689551 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:44.692389 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:44.696652 (+ 4263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:44.696662 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:44.696699 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:44.697275 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:44.697776 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:44.697783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:44.697790 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:44.698042 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:44.698746 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:44.698754 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:44.699218 (+ 464us) client_negotiation.cc:769] Sending connection context
0925 01:19:44.699569 (+ 351us) client_negotiation.cc:240] Negotiation successful
0925 01:19:44.700125 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3194,"thread_start_us":2957,"threads_started":1}
I20260925 01:19:44.703580 972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:44.673609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:55053 (local address 127.20.102.190:43283)
0925 01:19:44.680949 (+ 7340us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:44.680955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:44.680976 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:19:44.681036 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:44.681041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:44.681136 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:44.681248 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:44.683421 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:44.684951 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:44.687299 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:44.687939 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:44.688055 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:44.692708 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:44.692716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:44.692721 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:44.692766 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:44.696443 (+ 3677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:44.697459 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:44.697465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:44.697468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:44.697636 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:44.698199 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:44.698203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:44.698206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:44.698589 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:44.702802 (+ 4213us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:44.702950 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 01:19:44.703189 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7198,"thread_start_us":7093,"threads_started":1}
I20260925 01:19:44.705298 969 heartbeater.cc:347] Connected to a master server at 127.20.102.190:43283
I20260925 01:19:44.705653 969 heartbeater.cc:464] Registering TS with master...
I20260925 01:19:44.706566 969 heartbeater.cc:511] Master 127.20.102.190:43283 requested a full tablet report, sending...
I20260925 01:19:44.709224 755 ts_manager.cc:194] Registered new tserver with Master: 55dd0c09359d489abeaddf0f3580fd0b (127.20.102.129:41611)
I20260925 01:19:44.711316 755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:55053
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:19:44.769969 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.102.148:36949/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:43283
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:19:44 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299184, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
W20260925 01:19:45.252269 975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:19:45.252857 975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:19:45.253054 975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:19:45.264663 975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:19:45.264889 975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:19:45.265100 975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:19:45.280915 975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.20.102.148:36949/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:19:45.283171 975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:19:45.285136 975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:19:45.304752 986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:45.305764 987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:45.313776 992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:19:45.314447 975 server_base.cc:1034] running on GCE node
I20260925 01:19:45.315269 975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:19:45.316606 975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:19:45.317925 975 hybrid_clock.cc:648] HybridClock initialized: now 1790299185317855 us; error 56 us; skew 500 ppm
I20260925 01:19:45.318316 975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:19:45 dist-test-slave-s12s krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299185, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:45.322580 975 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:19:45.324590 975 webserver.cc:460] Webserver started at http://127.20.102.130:45833/ using document root <none> and password file <none>
I20260925 01:19:45.325554 975 fs_manager.cc:359] Metadata directory not provided
I20260925 01:19:45.325742 975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:19:45.326118 975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:19:45 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299185, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:19:45.348186 975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "1b819c035acb477f9d7b0719b6b79d05"
format_stamp: "Formatted at 2026-09-25 01:19:45 on dist-test-slave-s12s"
server_key: "6ac97fe7413641afbf9574ff1fbcae07"
server_key_iv: "eb38dd7ef4ba61a3e916e5f8a16e1a84"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:45.349424 975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "1b819c035acb477f9d7b0719b6b79d05"
format_stamp: "Formatted at 2026-09-25 01:19:45 on dist-test-slave-s12s"
server_key: "6ac97fe7413641afbf9574ff1fbcae07"
server_key_iv: "eb38dd7ef4ba61a3e916e5f8a16e1a84"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:45.356863 975 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260925 01:19:45.385387 999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:45.387431 975 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.000s
I20260925 01:19:45.387619 975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "1b819c035acb477f9d7b0719b6b79d05"
format_stamp: "Formatted at 2026-09-25 01:19:45 on dist-test-slave-s12s"
server_key: "6ac97fe7413641afbf9574ff1fbcae07"
server_key_iv: "eb38dd7ef4ba61a3e916e5f8a16e1a84"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:45.387893 975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:19:45.425482 975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:19:45.430910 975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:19:45.431248 975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:19:45.432439 975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:19:45.434244 975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:19:45.434336 975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:45.434501 975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:19:45.434588 975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:45.477977 975 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:34887
I20260925 01:19:45.478019 1111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:34887 every 8 connection(s)
I20260925 01:19:45.479676 975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:19:45.481017 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 975
I20260925 01:19:45.481215 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:19:45.481814 20890 external_mini_cluster.cc:1428] Setting key 40e355cd6b1c6b8595bf5ed53596842d
Sep 25 01:19:45 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299185, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
I20260925 01:19:45.517204 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:45.492227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:47027 (local address 127.20.102.190:43283)
0925 01:19:45.494001 (+ 1774us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:45.494008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:45.497532 (+ 3524us) server_negotiation.cc:407] Connection header received
0925 01:19:45.497631 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:45.497636 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:45.497720 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:45.497817 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:45.500174 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:45.501386 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:45.503784 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:45.504408 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:45.504505 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:45.509366 (+ 4861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:45.509373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:45.509377 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:45.509427 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:45.513097 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:45.514156 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:45.514162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:45.514166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:45.514317 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:45.514887 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:45.514893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:45.514896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:45.515305 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:45.515437 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:45.516536 (+ 1099us) server_negotiation.cc:299] Negotiation successful
0925 01:19:45.516820 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1641,"thread_start_us":1484,"threads_started":1}
I20260925 01:19:45.518399 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:45.491556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:43283 (local address 127.20.102.130:47027)
0925 01:19:45.494601 (+ 3045us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:45.494662 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:45.497133 (+ 2471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:45.498355 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:45.498368 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:45.499009 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:45.499953 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:45.499979 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:45.501635 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:45.501641 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:45.503609 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:45.503625 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:45.504911 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:45.504918 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:45.505216 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:45.506095 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:45.506126 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:45.508980 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:45.513315 (+ 4335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:45.513325 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:45.513345 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:45.514006 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:45.514457 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:45.514462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:45.514466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:45.514748 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:45.515558 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:45.515567 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:45.516045 (+ 478us) client_negotiation.cc:769] Sending connection context
0925 01:19:45.516374 (+ 329us) client_negotiation.cc:240] Negotiation successful
0925 01:19:45.516979 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2695,"thread_start_us":2455,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
I20260925 01:19:45.520255 1112 heartbeater.cc:347] Connected to a master server at 127.20.102.190:43283
I20260925 01:19:45.520578 1112 heartbeater.cc:464] Registering TS with master...
I20260925 01:19:45.521436 1112 heartbeater.cc:511] Master 127.20.102.190:43283 requested a full tablet report, sending...
I20260925 01:19:45.523041 755 ts_manager.cc:194] Registered new tserver with Master: 1b819c035acb477f9d7b0719b6b79d05 (127.20.102.130:34887)
I20260925 01:19:45.524097 755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:47027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:19:45.555866 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.102.148:36949/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:43283
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:19:45.715147 969 heartbeater.cc:503] Master 127.20.102.190:43283 was elected leader, sending a full tablet report...
W20260925 01:19:45.920871 1118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:19:45.921161 1118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:19:45.921254 1118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:19:45.928483 1118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:19:45.928591 1118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:19:45.928777 1118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:19:45.938594 1118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.20.102.148:36949/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:19:45.939882 1118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:19:45.941201 1118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:19:45.954453 1123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:45.954591 1126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:45.954653 1124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:19:45.955663 1118 server_base.cc:1034] running on GCE node
I20260925 01:19:45.956380 1118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:19:45.957476 1118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:19:45.958683 1118 hybrid_clock.cc:648] HybridClock initialized: now 1790299185958608 us; error 64 us; skew 500 ppm
I20260925 01:19:45.959029 1118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:19:45 dist-test-slave-s12s krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299185, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:45.962530 1118 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:19:45.964288 1118 webserver.cc:460] Webserver started at http://127.20.102.131:33559/ using document root <none> and password file <none>
I20260925 01:19:45.965132 1118 fs_manager.cc:359] Metadata directory not provided
I20260925 01:19:45.965296 1118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:19:45.965644 1118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 01:19:45 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299185, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:19:45.983361 1118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "1bcabff4ed6f4952bd091fba4ff087cd"
format_stamp: "Formatted at 2026-09-25 01:19:45 on dist-test-slave-s12s"
server_key: "49f5cb2e487f50dbbe6c531ac1d401a2"
server_key_iv: "b3efb7e04fb3823dba5074e491fe8733"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:45.984447 1118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "1bcabff4ed6f4952bd091fba4ff087cd"
format_stamp: "Formatted at 2026-09-25 01:19:45 on dist-test-slave-s12s"
server_key: "49f5cb2e487f50dbbe6c531ac1d401a2"
server_key_iv: "b3efb7e04fb3823dba5074e491fe8733"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:45.990253 1118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:19:46.005802 1133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:46.007345 1118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:19:46.007504 1118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "1bcabff4ed6f4952bd091fba4ff087cd"
format_stamp: "Formatted at 2026-09-25 01:19:45 on dist-test-slave-s12s"
server_key: "49f5cb2e487f50dbbe6c531ac1d401a2"
server_key_iv: "b3efb7e04fb3823dba5074e491fe8733"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:46.007718 1118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:19:46.020977 1118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:19:46.025370 1118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:19:46.025630 1118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:19:46.026539 1118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:19:46.027877 1118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:19:46.027951 1118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:46.028033 1118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:19:46.028077 1118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:46.063498 1118 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:44341
I20260925 01:19:46.063592 1245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:44341 every 8 connection(s)
I20260925 01:19:46.064929 1118 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:19:46.074558 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 1118
I20260925 01:19:46.074756 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:19:46.075284 20890 external_mini_cluster.cc:1428] Setting key 63dfe10462557af194467930ebfe2b88
Sep 25 01:19:46 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299185, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:19:46.085779 1248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:46.067650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:45443 (local address 127.20.102.190:43283)
0925 01:19:46.068083 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:46.068087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:46.069271 (+ 1184us) server_negotiation.cc:407] Connection header received
0925 01:19:46.071083 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:46.071089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:46.071199 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:46.071333 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:46.072830 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:46.073564 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:46.075234 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:46.075765 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:46.075907 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:46.079842 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:46.079848 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:46.079853 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:46.079899 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:46.082584 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:46.083338 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:46.083343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:46.083347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:46.083485 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:46.083941 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:46.083946 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:46.083949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:46.084231 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:46.084386 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:46.085161 (+ 775us) server_negotiation.cc:299] Negotiation successful
0925 01:19:46.085405 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":219,"threads_started":1}
I20260925 01:19:46.086725 1249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:46.068301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:43283 (local address 127.20.102.131:45443)
0925 01:19:46.069096 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:46.069139 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:46.070807 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:46.071573 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:46.071581 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:46.072026 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:46.072613 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:46.072630 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:46.073729 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:46.073736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:46.075079 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:46.075091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:46.075998 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:46.076005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:46.076298 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:46.077136 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:46.077161 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:46.079623 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:46.082743 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:46.082752 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:46.082769 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:46.083208 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:46.083598 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:46.083603 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:46.083607 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:46.083819 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:46.084363 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:46.084371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:46.084793 (+ 422us) client_negotiation.cc:769] Sending connection context
0925 01:19:46.085066 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 01:19:46.085590 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":368,"threads_started":1}
I20260925 01:19:46.088052 1246 heartbeater.cc:347] Connected to a master server at 127.20.102.190:43283
I20260925 01:19:46.088377 1246 heartbeater.cc:464] Registering TS with master...
I20260925 01:19:46.089283 1246 heartbeater.cc:511] Master 127.20.102.190:43283 requested a full tablet report, sending...
I20260925 01:19:46.091095 754 ts_manager.cc:194] Registered new tserver with Master: 1bcabff4ed6f4952bd091fba4ff087cd (127.20.102.131:44341)
I20260925 01:19:46.092125 754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:45443
I20260925 01:19:46.103747 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:19:46.105551 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 836
Sep 25 01:19:46 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:19:46.124615 1251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:46.106716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57286 (local address 127.20.102.129:41611)
0925 01:19:46.107332 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:46.107339 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:46.107365 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:19:46.107431 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:46.107437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:46.107681 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:46.107813 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:46.109646 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:46.110805 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:46.112464 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:46.113206 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:46.113373 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:46.118403 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:46.118418 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:46.118434 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:46.118474 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:46.121679 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:46.122404 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:46.122411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:46.122414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:46.122561 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:46.122892 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:46.122895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:46.122897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:46.123245 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:46.123483 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:46.123950 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 01:19:46.124174 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":320,"threads_started":1}
I20260925 01:19:46.126099 964 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:19:46.210264 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 836
I20260925 01:19:46.228639 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 975
Sep 25 01:19:46 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:19:46.245913 1253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:46.229722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43888 (local address 127.20.102.130:34887)
0925 01:19:46.230350 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:46.230357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:46.230385 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:19:46.230445 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:46.230451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:46.230672 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:46.230809 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:46.232045 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:46.233187 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:46.234794 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:46.235398 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:46.235538 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:46.240027 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:46.240040 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:46.240055 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:46.240122 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:46.243121 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:46.243863 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:46.243871 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:46.243873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:46.244020 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:46.244374 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:46.244377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:46.244380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:46.244877 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:46.245121 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:46.245393 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 01:19:46.245627 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":350,"threads_started":1}
I20260925 01:19:46.247421 1107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:19:46.332367 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 975
I20260925 01:19:46.348832 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 1118
Sep 25 01:19:46 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299182, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:19:46.365545 1255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:46.349891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.20.102.131:44341)
0925 01:19:46.350549 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:46.350556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:46.350583 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:19:46.350645 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:46.350650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:46.350855 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:46.350991 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:46.352240 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:46.353352 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:46.354630 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:46.355187 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:46.355293 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:46.359605 (+ 4312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:46.359616 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:46.359633 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:46.359727 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:46.362781 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:46.363387 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:46.363394 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:46.363396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:46.363522 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:46.363852 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:46.363855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:46.363857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:46.364210 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:46.364443 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:46.364916 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 01:19:46.365117 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":363,"threads_started":1}
I20260925 01:19:46.366827 1241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:19:46.454480 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 1118
I20260925 01:19:46.471271 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 723
I20260925 01:19:46.472615 785 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:19:46.580158 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 723
I20260925 01:19:46.602401 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:43283
--webserver_interface=127.20.102.190
--webserver_port=35469
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:43283
--ranger_config_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.102.148:36949/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.102.148:36949/kms with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:19:46.931460 1258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:19:46.931771 1258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:19:46.931928 1258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:19:46.939411 1258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:19:46.939533 1258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:19:46.939622 1258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:19:46.939735 1258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:19:46.939826 1258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:19:46.954041 1258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.102.148:36949/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:43283
--ranger_config_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:43283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=35469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:19:46.955416 1258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:19:46.956880 1258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:19:46.969157 1263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:46.969141 1266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:46.969221 1264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:19:46.970153 1258 server_base.cc:1034] running on GCE node
I20260925 01:19:46.970978 1258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:19:46.972505 1258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:19:46.973752 1258 hybrid_clock.cc:648] HybridClock initialized: now 1790299186973703 us; error 66 us; skew 500 ppm
I20260925 01:19:46.974118 1258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:19:46 dist-test-slave-s12s krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299186, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:46.977434 1258 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:19:46.979089 1258 webserver.cc:460] Webserver started at http://127.20.102.190:35469/ using document root <none> and password file <none>
I20260925 01:19:46.979787 1258 fs_manager.cc:359] Metadata directory not provided
I20260925 01:19:46.979940 1258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 01:19:47 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299186, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:19:47.035127 1258 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:19:47.038730 1274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:47.040333 1258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:19:47.040580 1258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "5245497731624901ac63a98a6db7f2a0"
format_stamp: "Formatted at 2026-09-25 01:19:42 on dist-test-slave-s12s"
server_key: "1747dbe48ceff461be158e1106491bc4"
server_key_iv: "ef2d0c098a87c7c664571efb6077b033"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:47.041870 1258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:19:47.070960 1258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:19:47.075451 1258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:19:47.075803 1258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:19:47.095618 1258 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:43283
I20260925 01:19:47.095685 1325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:43283 every 8 connection(s)
I20260925 01:19:47.097496 1258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:19:47.105108 1326 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:19:47.107493 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 1258
I20260925 01:19:47.108741 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.102.148:36949/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:41611
--local_ip_for_outbound_sockets=127.20.102.129
--tserver_master_addrs=127.20.102.190:43283
--webserver_port=42025
--webserver_interface=127.20.102.129
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.102.148:36949/kms with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:19:47.109735 1326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0: Bootstrap starting.
I20260925 01:19:47.132089 1326 log.cc:824] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0: Log is configured to *not* fsync() on all Append() calls
I20260925 01:19:47.142083 1326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:19:47.142732 1326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0: Bootstrap complete.
I20260925 01:19:47.152506 1326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5245497731624901ac63a98a6db7f2a0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 43283 } }
I20260925 01:19:47.153434 1326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5245497731624901ac63a98a6db7f2a0, State: Initialized, Role: FOLLOWER
I20260925 01:19:47.154105 1326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5245497731624901ac63a98a6db7f2a0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 43283 } }
I20260925 01:19:47.154361 1326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:19:47.154537 1326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:19:47.154800 1326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:19:47.158955 1326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5245497731624901ac63a98a6db7f2a0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 43283 } }
I20260925 01:19:47.159564 1326 leader_election.cc:302] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5245497731624901ac63a98a6db7f2a0; no voters:
I20260925 01:19:47.160264 1326 leader_election.cc:288] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 01:19:47.160425 1330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:19:47.161716 1330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [term 2 LEADER]: Becoming Leader. State: Replica: 5245497731624901ac63a98a6db7f2a0, State: Running, Role: LEADER
I20260925 01:19:47.162418 1330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5245497731624901ac63a98a6db7f2a0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 43283 } }
I20260925 01:19:47.162820 1326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:19:47.165186 1332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5245497731624901ac63a98a6db7f2a0. Latest consensus state: current_term: 2 leader_uuid: "5245497731624901ac63a98a6db7f2a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5245497731624901ac63a98a6db7f2a0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 43283 } } }
I20260925 01:19:47.165608 1332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:19:47.166159 1331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5245497731624901ac63a98a6db7f2a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5245497731624901ac63a98a6db7f2a0" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 43283 } } }
I20260925 01:19:47.166431 1331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:19:47.175761 1336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:19:47.181455 1336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:19:47.182832 1336 catalog_manager.cc:1259] Loaded cluster ID: d9bf2a24084e48b9a29eca9756a61b8c
I20260925 01:19:47.182992 1336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:19:47.187027 1336 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:19:47.188634 1336 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0: Loaded TSK: 0
I20260925 01:19:47.190188 1336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:19:47.193259 1326 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260925 01:19:47.524964 1328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:19:47.525302 1328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:19:47.525482 1328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:19:47.536971 1328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:19:47.537186 1328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:19:47.537405 1328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:19:47.551273 1328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.102.148:36949/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:41611
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=42025
--tserver_master_addrs=127.20.102.190:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:19:47.553282 1328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:19:47.554616 1328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:19:47.602547 1364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:47.603178 1363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:47.609157 1366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:19:47.612258 1328 server_base.cc:1034] running on GCE node
I20260925 01:19:47.613161 1328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:19:47.616472 1328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:19:47.617816 1328 hybrid_clock.cc:648] HybridClock initialized: now 1790299187617736 us; error 71 us; skew 500 ppm
I20260925 01:19:47.618218 1328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:19:47 dist-test-slave-s12s krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299187, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:47.622664 1328 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:19:47.627898 1328 webserver.cc:460] Webserver started at http://127.20.102.129:42025/ using document root <none> and password file <none>
I20260925 01:19:47.629029 1328 fs_manager.cc:359] Metadata directory not provided
I20260925 01:19:47.629287 1328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 01:19:47 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299187, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:19:47.663007 1328 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.000s
I20260925 01:19:47.667462 1374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:47.669627 1328 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260925 01:19:47.669873 1328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "55dd0c09359d489abeaddf0f3580fd0b"
format_stamp: "Formatted at 2026-09-25 01:19:44 on dist-test-slave-s12s"
server_key: "b06fe42eac09e9d322ecb69ec749995c"
server_key_iv: "d07aeca46580773e65fe7409fa4747fa"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:47.671614 1328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:19:47.697942 1328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:19:47.704120 1328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:19:47.704571 1328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:19:47.723290 1328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:19:47.737721 1328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:19:47.737957 1328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 01:19:47.738202 1328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:19:47.738371 1328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:47.813036 1328 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:41611
I20260925 01:19:47.813097 1486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:41611 every 8 connection(s)
I20260925 01:19:47.815171 1328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:19:47.818423 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 1328
I20260925 01:19:47.819365 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.102.148:36949/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:34887
--local_ip_for_outbound_sockets=127.20.102.130
--tserver_master_addrs=127.20.102.190:43283
--webserver_port=45833
--webserver_interface=127.20.102.130
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.102.148:36949/kms with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:19:47 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299187, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:19:47.861722 1489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:47.819185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:43283 (local address 127.20.102.129:34461)
0925 01:19:47.820153 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:47.820206 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:47.825077 (+ 4871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:47.831269 (+ 6192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:47.831285 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:47.832018 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:47.832954 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:47.832994 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:47.835613 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:47.835618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:47.837727 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:47.837744 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:47.839655 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:47.839664 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:47.839955 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:47.848188 (+ 8233us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:47.848225 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:47.851189 (+ 2964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:47.856096 (+ 4907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:47.856110 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:47.856131 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:47.856811 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:47.857365 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:47.857373 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:47.857378 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:47.857628 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:47.858747 (+ 1119us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:47.858757 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:47.859237 (+ 480us) client_negotiation.cc:769] Sending connection context
0925 01:19:47.859565 (+ 328us) client_negotiation.cc:240] Negotiation successful
0925 01:19:47.860176 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":1677,"threads_started":1}
I20260925 01:19:47.864380 1490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:47.820937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:34461 (local address 127.20.102.190:43283)
0925 01:19:47.829324 (+ 8387us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:47.829346 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:47.829408 (+ 62us) server_negotiation.cc:407] Connection header received
0925 01:19:47.829931 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:47.829955 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:47.830552 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:47.831064 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:47.833254 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:47.835400 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:47.837938 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:47.839014 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:47.839234 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:47.851611 (+ 12377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:47.851627 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:47.851651 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:47.851709 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:47.855863 (+ 4154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:47.856992 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:47.857001 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:47.857010 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:47.857223 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:47.857760 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:47.857765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:47.857769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:47.858455 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:47.858655 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:47.862521 (+ 3866us) server_negotiation.cc:299] Negotiation successful
0925 01:19:47.862983 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8061,"thread_start_us":7804,"threads_started":1}
I20260925 01:19:47.868604 1487 heartbeater.cc:347] Connected to a master server at 127.20.102.190:43283
I20260925 01:19:47.868997 1487 heartbeater.cc:464] Registering TS with master...
I20260925 01:19:47.869974 1487 heartbeater.cc:511] Master 127.20.102.190:43283 requested a full tablet report, sending...
W20260925 01:19:47.873775 1487 heartbeater.cc:644] Failed to heartbeat to 127.20.102.190:43283 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 01:19:47.874018 1487 heartbeater.cc:464] Registering TS with master...
I20260925 01:19:47.874320 1487 heartbeater.cc:511] Master 127.20.102.190:43283 requested a full tablet report, sending...
I20260925 01:19:47.875821 1487 heartbeater.cc:464] Registering TS with master...
I20260925 01:19:47.876124 1487 heartbeater.cc:511] Master 127.20.102.190:43283 requested a full tablet report, sending...
W20260925 01:19:47.877434 1487 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:19:48.300015 1491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:19:48.300382 1491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:19:48.300565 1491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:19:48.308785 1491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:19:48.308948 1491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:19:48.309171 1491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:19:48.319980 1491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.102.148:36949/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:34887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=45833
--tserver_master_addrs=127.20.102.190:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:19:48.321843 1491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:19:48.323599 1491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:19:48.342738 1506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:48.344033 1507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:48.344938 1509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:19:48.346014 1491 server_base.cc:1034] running on GCE node
I20260925 01:19:48.346727 1491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:19:48.347919 1491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:19:48.349138 1491 hybrid_clock.cc:648] HybridClock initialized: now 1790299188349088 us; error 60 us; skew 500 ppm
I20260925 01:19:48.349476 1491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:19:48 dist-test-slave-s12s krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299188, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:48.353086 1491 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:19:48.354815 1491 webserver.cc:460] Webserver started at http://127.20.102.130:45833/ using document root <none> and password file <none>
I20260925 01:19:48.355661 1491 fs_manager.cc:359] Metadata directory not provided
I20260925 01:19:48.355831 1491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 01:19:48 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299188, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:19:48.396821 1491 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.006s sys 0.005s
I20260925 01:19:48.400374 1519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:48.402405 1491 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:19:48.402626 1491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "1b819c035acb477f9d7b0719b6b79d05"
format_stamp: "Formatted at 2026-09-25 01:19:45 on dist-test-slave-s12s"
server_key: "6ac97fe7413641afbf9574ff1fbcae07"
server_key_iv: "eb38dd7ef4ba61a3e916e5f8a16e1a84"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:48.404085 1491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:19:48.451586 1491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:19:48.456427 1491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:19:48.456842 1491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:19:48.458076 1491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:19:48.459766 1491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:19:48.459905 1491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:48.460063 1491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:19:48.460162 1491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 01:19:48 dist-test-slave-s12s krb5kdc[32747](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790299188, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:48.550189 1491 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:34887
I20260925 01:19:48.550241 1631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:34887 every 8 connection(s)
I20260925 01:19:48.551690 1491 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:19:48.555754 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 1491
I20260925 01:19:48.556576 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.20.102.148:36949/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:44341
--local_ip_for_outbound_sockets=127.20.102.131
--tserver_master_addrs=127.20.102.190:43283
--webserver_port=33559
--webserver_interface=127.20.102.131
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:39107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.20.102.148:36949/kms with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:19:48 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299188, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:19:48.586428 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:48.555464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:43283 (local address 127.20.102.130:36019)
0925 01:19:48.556391 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:48.556449 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:48.559029 (+ 2580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:48.560239 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:48.560248 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:48.560840 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:48.561493 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:48.561513 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:48.562706 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:48.562710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:48.564425 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:48.564442 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:48.565673 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:48.565678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:48.565887 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:48.566836 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:48.566861 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:48.569406 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:48.581025 (+ 11619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:48.581039 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:48.581063 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:48.581827 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:48.582372 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:48.582377 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:48.582380 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:48.582699 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:48.583456 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:48.583465 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:48.583988 (+ 523us) client_negotiation.cc:769] Sending connection context
0925 01:19:48.584300 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 01:19:48.584942 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":378,"threads_started":1}
I20260925 01:19:48.588399 1634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:48.554666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:36019 (local address 127.20.102.190:43283)
0925 01:19:48.555246 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:48.555254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:48.556585 (+ 1331us) server_negotiation.cc:407] Connection header received
0925 01:19:48.559481 (+ 2896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:48.559488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:48.559586 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:48.559675 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:48.561695 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:48.562476 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:48.564619 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:48.565285 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:48.565370 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:48.569669 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:48.569678 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:48.569685 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:48.569749 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:48.580760 (+ 11011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:48.582040 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:48.582045 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:48.582049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:48.582217 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:48.582876 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:48.582882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:48.582885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:48.583290 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:48.587615 (+ 4325us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:48.587768 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 01:19:48.588032 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":288,"threads_started":1}
I20260925 01:19:48.590206 1632 heartbeater.cc:347] Connected to a master server at 127.20.102.190:43283
I20260925 01:19:48.590581 1632 heartbeater.cc:464] Registering TS with master...
I20260925 01:19:48.591513 1632 heartbeater.cc:511] Master 127.20.102.190:43283 requested a full tablet report, sending...
I20260925 01:19:48.593786 1291 ts_manager.cc:194] Registered new tserver with Master: 1b819c035acb477f9d7b0719b6b79d05 (127.20.102.130:34887)
I20260925 01:19:48.596093 1291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:36019
I20260925 01:19:48.877586 1487 heartbeater.cc:464] Registering TS with master...
I20260925 01:19:48.878065 1487 heartbeater.cc:511] Master 127.20.102.190:43283 requested a full tablet report, sending...
I20260925 01:19:48.879372 1291 ts_manager.cc:194] Registered new tserver with Master: 55dd0c09359d489abeaddf0f3580fd0b (127.20.102.129:41611)
I20260925 01:19:48.880582 1291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:34461
Sep 25 01:19:48 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299188, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
W20260925 01:19:49.024397 1636 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:19:49.024755 1636 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:19:49.024936 1636 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:19:49.035679 1636 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:19:49.035820 1636 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:19:49.035987 1636 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:19:49.050025 1636 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.20.102.148:36949/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:44341
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=33559
--tserver_master_addrs=127.20.102.190:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:19:49.051730 1636 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:19:49.053669 1636 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:19:49.070025 1642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:49.070526 1643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:49.086476 1645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:19:49.086941 1636 server_base.cc:1034] running on GCE node
I20260925 01:19:49.087782 1636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:19:49.089282 1636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:19:49.090557 1636 hybrid_clock.cc:648] HybridClock initialized: now 1790299189090497 us; error 46 us; skew 500 ppm
I20260925 01:19:49.090912 1636 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:19:49 dist-test-slave-s12s krb5kdc[32747](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299189, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:49.094303 1636 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:19:49.113286 1636 webserver.cc:460] Webserver started at http://127.20.102.131:33559/ using document root <none> and password file <none>
I20260925 01:19:49.114121 1636 fs_manager.cc:359] Metadata directory not provided
I20260925 01:19:49.114298 1636 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 01:19:49 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299189, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:19:49.165097 1636 fs_manager.cc:709] Time spent opening directory manager: real 0.021s user 0.008s sys 0.012s
I20260925 01:19:49.182629 1653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:49.185195 1636 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.008s sys 0.008s
I20260925 01:19:49.185405 1636 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "1bcabff4ed6f4952bd091fba4ff087cd"
format_stamp: "Formatted at 2026-09-25 01:19:45 on dist-test-slave-s12s"
server_key: "49f5cb2e487f50dbbe6c531ac1d401a2"
server_key_iv: "b3efb7e04fb3823dba5074e491fe8733"
server_key_version: "kuduclusterkey@0"
I20260925 01:19:49.186898 1636 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 25 01:19:49 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299188, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for HTTP/127.20.102.148@KRBTEST.COM
I20260925 01:19:49.236479 1636 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:19:49.241911 1636 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:19:49.242221 1636 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:19:49.249692 1636 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:19:49.258100 1636 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:19:49.258239 1636 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:49.258358 1636 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:19:49.258431 1636 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:49.428228 1636 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:44341
I20260925 01:19:49.428241 1766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:44341 every 8 connection(s)
I20260925 01:19:49.430166 1636 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:19:49.441327 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 1636
Sep 25 01:19:49 dist-test-slave-s12s krb5kdc[32747](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299189, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:19:49.473402 1770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.433927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:43283 (local address 127.20.102.131:58107)
0925 01:19:49.449201 (+ 15274us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:49.449265 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:49.451074 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:49.452525 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:49.452538 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:49.453235 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:49.454111 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:49.454138 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.455722 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.455728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:49.457829 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:49.457844 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.459130 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.459138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:49.459423 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.460233 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:49.460263 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:49.462926 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:49.468080 (+ 5154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:49.468091 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:49.468109 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:49.468745 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:49.469232 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:49.469238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:49.469241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:49.469503 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:49.470256 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:49.470264 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:49.470707 (+ 443us) client_negotiation.cc:769] Sending connection context
0925 01:19:49.471241 (+ 534us) client_negotiation.cc:240] Negotiation successful
0925 01:19:49.471819 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14941,"thread_start_us":14680,"threads_started":1}
I20260925 01:19:49.475238 1769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.433195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:58107 (local address 127.20.102.190:43283)
0925 01:19:49.451845 (+ 18650us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:49.451853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:49.451879 (+ 26us) server_negotiation.cc:407] Connection header received
0925 01:19:49.451956 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:49.451963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:49.452059 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:49.452183 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:49.454338 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.455473 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.458008 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.458629 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.458745 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.463183 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:49.463190 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:49.463195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:49.463232 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:49.467902 (+ 4670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:49.468913 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:49.468918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:49.468922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:49.469075 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:49.469646 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:49.469651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:49.469655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:49.470030 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:49.470173 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:49.474483 (+ 4310us) server_negotiation.cc:299] Negotiation successful
0925 01:19:49.474739 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15218,"thread_start_us":18153,"threads_started":1}
I20260925 01:19:49.476382 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.451557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48728 (local address 127.20.102.190:43283)
0925 01:19:49.452104 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:49.452112 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:49.452129 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:19:49.452173 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:49.452177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:49.452259 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:49.453628 (+ 1369us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:49.456850 (+ 3222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.458070 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.460059 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.460776 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.460895 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.465558 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:49.465565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:49.465570 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:49.465625 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:49.472434 (+ 6809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:49.473303 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:49.473309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:49.473312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:49.473478 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:49.474559 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:49.474565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:49.474568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:49.474925 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:49.475201 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:49.475786 (+ 585us) server_negotiation.cc:299] Negotiation successful
0925 01:19:49.476000 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":806,"threads_started":1}
I20260925 01:19:49.476609 1767 heartbeater.cc:347] Connected to a master server at 127.20.102.190:43283
I20260925 01:19:49.477012 1767 heartbeater.cc:464] Registering TS with master...
I20260925 01:19:49.477880 1767 heartbeater.cc:511] Master 127.20.102.190:43283 requested a full tablet report, sending...
I20260925 01:19:49.479636 1291 ts_manager.cc:194] Registered new tserver with Master: 1bcabff4ed6f4952bd091fba4ff087cd (127.20.102.131:44341)
I20260925 01:19:49.480373 1291 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:58107
I20260925 01:19:49.491807 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:19:49.508522 1772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.495370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48734 (local address 127.20.102.190:43283)
0925 01:19:49.495641 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:49.495647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:49.495886 (+ 239us) server_negotiation.cc:407] Connection header received
0925 01:19:49.496047 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:49.496050 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:49.496134 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:49.496216 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:49.497995 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.498793 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.500350 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.500837 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.500941 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.503498 (+ 2557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:49.503503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:49.503506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:49.503536 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:49.506056 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:49.506762 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:49.506766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:49.506768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:49.506878 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:49.507287 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:49.507290 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:49.507292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:49.507536 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:49.507718 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:49.508010 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 01:19:49.508179 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260925 01:19:49.513815 1291 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:19:49.521319 1291 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:19:49.559538 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.536305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43890 (local address 127.20.102.130:34887)
0925 01:19:49.549222 (+ 12917us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:49.549232 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:49.549263 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:19:49.549382 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:49.549391 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:49.549663 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:49.550650 (+ 987us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:19:49.550880 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.552631 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.555513 (+ 2882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.557044 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.558766 (+ 1722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.558942 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:49.559013 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 01:19:49.559194 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12750,"thread_start_us":12625,"threads_started":1}
I20260925 01:19:49.560168 1789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.536015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:34887 (local address 127.0.0.1:43890)
0925 01:19:49.548158 (+ 12143us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:49.548181 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:49.548314 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:49.549924 (+ 1610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:49.549931 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:49.549965 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:19:49.550415 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:49.550426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.552813 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.552817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:49.555337 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:49.555352 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.558074 (+ 2722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.558083 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:49.558321 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.558388 (+ 67us) client_negotiation.cc:769] Sending connection context
0925 01:19:49.558496 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 01:19:49.558696 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11972,"thread_start_us":11005,"threads_started":1}
I20260925 01:19:49.563755 1790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.535865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:44341 (local address 127.0.0.1:56876)
0925 01:19:49.547421 (+ 11556us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:49.547469 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:49.547655 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:49.550424 (+ 2769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:49.550428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:49.550455 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:19:49.550815 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:49.550827 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.552800 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.552805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:49.558983 (+ 6178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:49.558997 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.562763 (+ 3766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.562771 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:49.563039 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.563108 (+ 69us) client_negotiation.cc:769] Sending connection context
0925 01:19:49.563190 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 01:19:49.563318 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11300,"mutex_wait_us":47,"thread_start_us":11962,"threads_started":1}
I20260925 01:19:49.563773 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.536327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56876 (local address 127.20.102.131:44341)
0925 01:19:49.549538 (+ 13211us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:49.549546 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:49.549588 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:19:49.549679 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:49.549687 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:49.549932 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:49.550120 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:19:49.551016 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.552601 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.559182 (+ 6581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.561201 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.561407 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.561598 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:49.563239 (+ 1641us) server_negotiation.cc:299] Negotiation successful
0925 01:19:49.563401 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13054,"thread_start_us":12806,"threads_started":1}
I20260925 01:19:49.564385 1567 tablet_service.cc:1467] Processing CreateTablet for tablet 98734ccc4d0d408984ef9a1eca2e27ba (DEFAULT_TABLE table=test-table [id=ba4189243ccb4600a5c0427e8050bf16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:19:49.565907 1794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.554205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:41611 (local address 127.0.0.1:57292)
0925 01:19:49.557981 (+ 3776us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:49.558001 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:49.558137 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:49.559215 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:49.559223 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:49.559254 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:19:49.559702 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:49.559715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.561476 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.561481 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:49.562985 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:49.563001 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.565071 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.565090 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:49.565266 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.565303 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 01:19:49.565397 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 01:19:49.565533 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3609,"mutex_wait_us":23,"thread_start_us":3430,"threads_started":1}
I20260925 01:19:49.566540 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.548647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57292 (local address 127.20.102.129:41611)
0925 01:19:49.557611 (+ 8964us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:49.557623 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:49.558285 (+ 662us) server_negotiation.cc:407] Connection header received
0925 01:19:49.558428 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:49.558437 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:49.558704 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:49.558925 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:19:49.559960 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.561335 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.563172 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.564827 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.565050 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.565223 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:49.565485 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 01:19:49.565653 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8778,"thread_start_us":8567,"threads_started":1}
I20260925 01:19:49.567097 1567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98734ccc4d0d408984ef9a1eca2e27ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:19:49.567484 1702 tablet_service.cc:1467] Processing CreateTablet for tablet 98734ccc4d0d408984ef9a1eca2e27ba (DEFAULT_TABLE table=test-table [id=ba4189243ccb4600a5c0427e8050bf16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:19:49.568771 1702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98734ccc4d0d408984ef9a1eca2e27ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:19:49.570606 1422 tablet_service.cc:1467] Processing CreateTablet for tablet 98734ccc4d0d408984ef9a1eca2e27ba (DEFAULT_TABLE table=test-table [id=ba4189243ccb4600a5c0427e8050bf16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:19:49.572024 1422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98734ccc4d0d408984ef9a1eca2e27ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:19:49.581144 1795 tablet_bootstrap.cc:492] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd: Bootstrap starting.
I20260925 01:19:49.583702 1796 tablet_bootstrap.cc:492] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05: Bootstrap starting.
I20260925 01:19:49.585173 1795 tablet_bootstrap.cc:654] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd: Neither blocks nor log segments found. Creating new log.
I20260925 01:19:49.586519 1795 log.cc:824] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd: Log is configured to *not* fsync() on all Append() calls
I20260925 01:19:49.587234 1796 tablet_bootstrap.cc:654] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05: Neither blocks nor log segments found. Creating new log.
I20260925 01:19:49.587900 1797 tablet_bootstrap.cc:492] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b: Bootstrap starting.
I20260925 01:19:49.588616 1796 log.cc:824] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05: Log is configured to *not* fsync() on all Append() calls
I20260925 01:19:49.590698 1795 tablet_bootstrap.cc:492] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd: No bootstrap required, opened a new log
I20260925 01:19:49.590972 1795 ts_tablet_manager.cc:1397] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 01:19:49.591688 1796 tablet_bootstrap.cc:492] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05: No bootstrap required, opened a new log
I20260925 01:19:49.591993 1796 ts_tablet_manager.cc:1397] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260925 01:19:49.591996 1797 tablet_bootstrap.cc:654] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b: Neither blocks nor log segments found. Creating new log.
I20260925 01:19:49.593358 1797 log.cc:824] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:19:49.596338 1797 tablet_bootstrap.cc:492] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b: No bootstrap required, opened a new log
I20260925 01:19:49.596590 1797 ts_tablet_manager.cc:1397] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 01:19:49.598995 1795 raft_consensus.cc:348] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcabff4ed6f4952bd091fba4ff087cd" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44341 } } peers { permanent_uuid: "1b819c035acb477f9d7b0719b6b79d05" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34887 } } peers { permanent_uuid: "55dd0c09359d489abeaddf0f3580fd0b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41611 } }
I20260925 01:19:49.599443 1795 raft_consensus.cc:374] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:19:49.599538 1795 raft_consensus.cc:729] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bcabff4ed6f4952bd091fba4ff087cd, State: Initialized, Role: FOLLOWER
I20260925 01:19:49.600176 1795 consensus_queue.cc:260] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcabff4ed6f4952bd091fba4ff087cd" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44341 } } peers { permanent_uuid: "1b819c035acb477f9d7b0719b6b79d05" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34887 } } peers { permanent_uuid: "55dd0c09359d489abeaddf0f3580fd0b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41611 } }
I20260925 01:19:49.601275 1632 heartbeater.cc:503] Master 127.20.102.190:43283 was elected leader, sending a full tablet report...
I20260925 01:19:49.601459 1767 heartbeater.cc:503] Master 127.20.102.190:43283 was elected leader, sending a full tablet report...
I20260925 01:19:49.602177 1795 ts_tablet_manager.cc:1428] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260925 01:19:49.603972 1796 raft_consensus.cc:348] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcabff4ed6f4952bd091fba4ff087cd" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44341 } } peers { permanent_uuid: "1b819c035acb477f9d7b0719b6b79d05" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34887 } } peers { permanent_uuid: "55dd0c09359d489abeaddf0f3580fd0b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41611 } }
I20260925 01:19:49.604375 1796 raft_consensus.cc:374] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:19:49.604473 1796 raft_consensus.cc:729] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b819c035acb477f9d7b0719b6b79d05, State: Initialized, Role: FOLLOWER
I20260925 01:19:49.605170 1796 consensus_queue.cc:260] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcabff4ed6f4952bd091fba4ff087cd" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44341 } } peers { permanent_uuid: "1b819c035acb477f9d7b0719b6b79d05" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34887 } } peers { permanent_uuid: "55dd0c09359d489abeaddf0f3580fd0b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41611 } }
I20260925 01:19:49.606889 1796 ts_tablet_manager.cc:1428] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05: Time spent starting tablet: real 0.015s user 0.009s sys 0.003s
I20260925 01:19:49.606954 1797 raft_consensus.cc:348] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcabff4ed6f4952bd091fba4ff087cd" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44341 } } peers { permanent_uuid: "1b819c035acb477f9d7b0719b6b79d05" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34887 } } peers { permanent_uuid: "55dd0c09359d489abeaddf0f3580fd0b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41611 } }
I20260925 01:19:49.607372 1797 raft_consensus.cc:374] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:19:49.607501 1797 raft_consensus.cc:729] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55dd0c09359d489abeaddf0f3580fd0b, State: Initialized, Role: FOLLOWER
I20260925 01:19:49.608085 1797 consensus_queue.cc:260] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcabff4ed6f4952bd091fba4ff087cd" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44341 } } peers { permanent_uuid: "1b819c035acb477f9d7b0719b6b79d05" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34887 } } peers { permanent_uuid: "55dd0c09359d489abeaddf0f3580fd0b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41611 } }
I20260925 01:19:49.609196 1487 heartbeater.cc:503] Master 127.20.102.190:43283 was elected leader, sending a full tablet report...
I20260925 01:19:49.609931 1797 ts_tablet_manager.cc:1428] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 01:19:49.672067 1802 raft_consensus.cc:482] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:19:49.672411 1802 raft_consensus.cc:504] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcabff4ed6f4952bd091fba4ff087cd" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44341 } } peers { permanent_uuid: "1b819c035acb477f9d7b0719b6b79d05" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34887 } } peers { permanent_uuid: "55dd0c09359d489abeaddf0f3580fd0b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41611 } }
I20260925 01:19:49.674309 1802 leader_election.cc:288] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1bcabff4ed6f4952bd091fba4ff087cd (127.20.102.131:44341), 55dd0c09359d489abeaddf0f3580fd0b (127.20.102.129:41611)
I20260925 01:19:49.681523 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.674816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:45655 (local address 127.20.102.131:44341)
0925 01:19:49.674983 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:49.674988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:49.675634 (+ 646us) server_negotiation.cc:407] Connection header received
0925 01:19:49.675902 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:49.675911 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:49.676001 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:49.676066 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:19:49.676875 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.677695 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.679759 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.680545 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.680731 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.680812 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:49.681131 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 01:19:49.681256 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:19:49.681659 1804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.674721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:44341 (local address 127.20.102.130:45655)
0925 01:19:49.675501 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:49.675521 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:49.675694 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:49.676257 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:49.676261 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:49.676282 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:19:49.676723 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:49.676738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.677824 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.677832 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:49.679601 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:49.679617 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.680774 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.680780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:49.680968 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.680997 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 01:19:49.681107 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 01:19:49.681230 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"spinlock_wait_cycles":24448,"thread_start_us":415,"threads_started":1}
I20260925 01:19:49.681932 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.675883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:47975 (local address 127.20.102.129:41611)
0925 01:19:49.676124 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:49.676130 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:49.676350 (+ 220us) server_negotiation.cc:407] Connection header received
0925 01:19:49.676619 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:49.676625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:49.676729 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:49.676831 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:19:49.677364 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.678190 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.679998 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.680819 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.681014 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.681099 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:49.681375 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 01:19:49.681500 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 01:19:49.681990 1805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.675660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:41611 (local address 127.20.102.130:47975)
0925 01:19:49.676243 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:49.676258 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:49.676435 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:49.676847 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:49.676850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:49.676873 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:19:49.677204 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:49.677212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.678339 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.678345 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:49.679845 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:49.679857 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.681051 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.681057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:49.681206 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.681229 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 01:19:49.681339 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 01:19:49.681463 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":269,"threads_started":1}
W20260925 01:19:49.682232 1768 tablet.cc:2369] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:19:49.683065 1722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98734ccc4d0d408984ef9a1eca2e27ba" candidate_uuid: "1b819c035acb477f9d7b0719b6b79d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcabff4ed6f4952bd091fba4ff087cd" is_pre_election: true
I20260925 01:19:49.683169 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98734ccc4d0d408984ef9a1eca2e27ba" candidate_uuid: "1b819c035acb477f9d7b0719b6b79d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55dd0c09359d489abeaddf0f3580fd0b" is_pre_election: true
I20260925 01:19:49.683655 1722 raft_consensus.cc:2454] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b819c035acb477f9d7b0719b6b79d05 in term 0.
I20260925 01:19:49.683700 1442 raft_consensus.cc:2454] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b819c035acb477f9d7b0719b6b79d05 in term 0.
I20260925 01:19:49.684329 1522 leader_election.cc:302] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b819c035acb477f9d7b0719b6b79d05, 1bcabff4ed6f4952bd091fba4ff087cd; no voters:
I20260925 01:19:49.684659 1802 raft_consensus.cc:2781] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:19:49.684875 1802 raft_consensus.cc:482] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:19:49.684995 1802 raft_consensus.cc:3037] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:19:49.687220 1802 raft_consensus.cc:504] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcabff4ed6f4952bd091fba4ff087cd" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44341 } } peers { permanent_uuid: "1b819c035acb477f9d7b0719b6b79d05" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34887 } } peers { permanent_uuid: "55dd0c09359d489abeaddf0f3580fd0b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41611 } }
I20260925 01:19:49.687799 1802 leader_election.cc:288] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [CANDIDATE]: Term 1 election: Requested vote from peers 1bcabff4ed6f4952bd091fba4ff087cd (127.20.102.131:44341), 55dd0c09359d489abeaddf0f3580fd0b (127.20.102.129:41611)
I20260925 01:19:49.688335 1722 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98734ccc4d0d408984ef9a1eca2e27ba" candidate_uuid: "1b819c035acb477f9d7b0719b6b79d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bcabff4ed6f4952bd091fba4ff087cd"
I20260925 01:19:49.688421 1442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98734ccc4d0d408984ef9a1eca2e27ba" candidate_uuid: "1b819c035acb477f9d7b0719b6b79d05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55dd0c09359d489abeaddf0f3580fd0b"
I20260925 01:19:49.688531 1722 raft_consensus.cc:3037] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:19:49.688678 1442 raft_consensus.cc:3037] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:19:49.691022 1722 raft_consensus.cc:2454] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b819c035acb477f9d7b0719b6b79d05 in term 1.
I20260925 01:19:49.691380 1522 leader_election.cc:302] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b819c035acb477f9d7b0719b6b79d05, 1bcabff4ed6f4952bd091fba4ff087cd; no voters:
I20260925 01:19:49.691529 1442 raft_consensus.cc:2454] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b819c035acb477f9d7b0719b6b79d05 in term 1.
I20260925 01:19:49.691725 1802 raft_consensus.cc:2781] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:19:49.692276 1802 raft_consensus.cc:686] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [term 1 LEADER]: Becoming Leader. State: Replica: 1b819c035acb477f9d7b0719b6b79d05, State: Running, Role: LEADER
I20260925 01:19:49.692864 1802 consensus_queue.cc:237] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcabff4ed6f4952bd091fba4ff087cd" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44341 } } peers { permanent_uuid: "1b819c035acb477f9d7b0719b6b79d05" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34887 } } peers { permanent_uuid: "55dd0c09359d489abeaddf0f3580fd0b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41611 } }
I20260925 01:19:49.697470 1291 catalog_manager.cc:5495] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b819c035acb477f9d7b0719b6b79d05 (127.20.102.130). New cstate: current_term: 1 leader_uuid: "1b819c035acb477f9d7b0719b6b79d05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1bcabff4ed6f4952bd091fba4ff087cd" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b819c035acb477f9d7b0719b6b79d05" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55dd0c09359d489abeaddf0f3580fd0b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41611 } health_report { overall_health: UNKNOWN } } }
I20260925 01:19:49.739650 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.734438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43898 (local address 127.20.102.130:34887)
0925 01:19:49.734624 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:49.734629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:49.734646 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:19:49.734834 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:49.734838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:49.734916 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:49.735039 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:19:49.735401 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.736195 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.737716 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.738170 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.738263 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.738692 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:19:49.739175 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:19:49.739280 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:49.739369 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 01:19:49.739449 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 01:19:49.747929 1722 raft_consensus.cc:1261] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd [term 1 FOLLOWER]: Refusing update from remote peer 1b819c035acb477f9d7b0719b6b79d05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:19:49.747929 1442 raft_consensus.cc:1261] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b [term 1 FOLLOWER]: Refusing update from remote peer 1b819c035acb477f9d7b0719b6b79d05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:19:49.748965 1806 consensus_queue.cc:1035] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1bcabff4ed6f4952bd091fba4ff087cd" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:19:49.749321 1802 consensus_queue.cc:1035] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55dd0c09359d489abeaddf0f3580fd0b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:19:49.760515 1809 mvcc.cc:205] Tried to move back new op lower bound from 7333065481196929024 to 7333065480985825280. Current Snapshot: MvccSnapshot[applied={T|T < 7333065481196929024}]
I20260925 01:19:49.773977 1291 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:19:49.774279 1291 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:19:49.778083 1291 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5245497731624901ac63a98a6db7f2a0: Sending DeleteTablet for 3 replicas of tablet 98734ccc4d0d408984ef9a1eca2e27ba
I20260925 01:19:49.779342 1567 tablet_service.cc:1514] Processing DeleteTablet for tablet 98734ccc4d0d408984ef9a1eca2e27ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:19:49 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:43890
I20260925 01:19:49.779351 1702 tablet_service.cc:1514] Processing DeleteTablet for tablet 98734ccc4d0d408984ef9a1eca2e27ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:19:49 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:56876
I20260925 01:19:49.779387 1422 tablet_service.cc:1514] Processing DeleteTablet for tablet 98734ccc4d0d408984ef9a1eca2e27ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:19:49 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:57292
I20260925 01:19:49.785708 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 1328
I20260925 01:19:49.795109 1820 tablet_replica.cc:330] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd: stopping tablet replica
I20260925 01:19:49.795603 1820 raft_consensus.cc:2229] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:19:49.796169 1820 raft_consensus.cc:2258] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:19:49.798601 1820 ts_tablet_manager.cc:1905] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:19:49.805660 1820 ts_tablet_manager.cc:1918] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:19:49.805828 1820 log.cc:1196] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/98734ccc4d0d408984ef9a1eca2e27ba
I20260925 01:19:49.806376 1820 ts_tablet_manager.cc:1939] T 98734ccc4d0d408984ef9a1eca2e27ba P 1bcabff4ed6f4952bd091fba4ff087cd: Deleting consensus metadata
I20260925 01:19:49.808321 1277 catalog_manager.cc:4841] TS 1bcabff4ed6f4952bd091fba4ff087cd (127.20.102.131:44341): tablet 98734ccc4d0d408984ef9a1eca2e27ba (table test-table [id=ba4189243ccb4600a5c0427e8050bf16]) successfully deleted
I20260925 01:19:49.809295 1818 tablet_replica.cc:330] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05: stopping tablet replica
I20260925 01:19:49.809725 1818 raft_consensus.cc:2229] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:19:49.810472 1818 raft_consensus.cc:2258] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:19:49.811226 1819 tablet_replica.cc:330] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b: stopping tablet replica
I20260925 01:19:49.811599 1819 raft_consensus.cc:2229] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:19:49.812095 1819 raft_consensus.cc:2258] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:19:49.814211 1818 ts_tablet_manager.cc:1905] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:19:49.814357 1819 ts_tablet_manager.cc:1905] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:19:49.821339 1819 ts_tablet_manager.cc:1918] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:19:49.821493 1819 log.cc:1196] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/98734ccc4d0d408984ef9a1eca2e27ba
I20260925 01:19:49.821859 1818 ts_tablet_manager.cc:1918] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:19:49.822062 1819 ts_tablet_manager.cc:1939] T 98734ccc4d0d408984ef9a1eca2e27ba P 55dd0c09359d489abeaddf0f3580fd0b: Deleting consensus metadata
I20260925 01:19:49.822053 1818 log.cc:1196] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/98734ccc4d0d408984ef9a1eca2e27ba
I20260925 01:19:49.822630 1818 ts_tablet_manager.cc:1939] T 98734ccc4d0d408984ef9a1eca2e27ba P 1b819c035acb477f9d7b0719b6b79d05: Deleting consensus metadata
I20260925 01:19:49.824014 1277 catalog_manager.cc:4841] TS 55dd0c09359d489abeaddf0f3580fd0b (127.20.102.129:41611): tablet 98734ccc4d0d408984ef9a1eca2e27ba (table test-table [id=ba4189243ccb4600a5c0427e8050bf16]) successfully deleted
I20260925 01:19:49.824362 1275 catalog_manager.cc:4841] TS 1b819c035acb477f9d7b0719b6b79d05 (127.20.102.130:34887): tablet 98734ccc4d0d408984ef9a1eca2e27ba (table test-table [id=ba4189243ccb4600a5c0427e8050bf16]) successfully deleted
I20260925 01:19:49.829144 1793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.787391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57296 (local address 127.20.102.129:41611)
0925 01:19:49.787552 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:49.787558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:49.787580 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:19:49.787750 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:49.787756 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:49.787833 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:49.787957 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:49.796658 (+ 8701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.798268 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.801292 (+ 3024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.802259 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.802415 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.821736 (+ 19321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:49.821750 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:49.821772 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:49.821830 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:49.826001 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:49.826755 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:49.826764 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:49.826767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:49.826917 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:49.827325 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:49.827330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:49.827333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:49.827719 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:49.828060 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:49.828371 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 01:19:49.828611 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 01:19:49.829772 1482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:19:49.919765 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 1328
I20260925 01:19:49.938107 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 1491
I20260925 01:19:49.962152 1791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:49.938943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43908 (local address 127.20.102.130:34887)
0925 01:19:49.939133 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:49.939137 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:49.939269 (+ 132us) server_negotiation.cc:407] Connection header received
0925 01:19:49.939465 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:49.939469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:49.939555 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:49.939642 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:49.941668 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.942485 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.944554 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:49.945263 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:49.945370 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:49.954697 (+ 9327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:49.954714 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:49.954732 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:49.954771 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:49.958522 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:49.959439 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:49.959447 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:49.959449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:49.959642 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:49.960117 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:49.960120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:49.960122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:49.960572 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:49.961120 (+ 548us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:49.961480 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 01:19:49.961739 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:19:49.962993 1627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:19:50.079618 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 1491
I20260925 01:19:50.098901 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 1636
I20260925 01:19:50.114818 1792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:50.099769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.20.102.131:44341)
0925 01:19:50.100237 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:50.100241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:50.100256 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:19:50.100308 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:50.100311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:50.100369 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:50.100480 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:50.101813 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:50.102614 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:50.104850 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:50.105247 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:50.105347 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:50.108761 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:50.108775 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:50.108791 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:50.108839 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:50.111602 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:50.112495 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:50.112502 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:50.112505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:50.112752 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:50.113231 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:50.113235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:50.113237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:50.113651 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:50.113879 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:50.114264 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 01:19:50.114471 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361}
I20260925 01:19:50.115579 1762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:19:50.204586 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 1636
I20260925 01:19:50.222210 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 1258
I20260925 01:19:50.223414 1321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:19:50.337347 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 1258
I20260925 01:19:50.365036 20890 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 01:19:50.700543 20890 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 01:19:50.700829 20890 mini_ranger.cc:66] Stopping Ranger...
I20260925 01:19:51.046259 20890 mini_ranger.cc:68] Stopped Ranger
2026-09-25 01:19:51.046 UTC [309] LOG: received smart shutdown request
2026-09-25 01:19:51.049 UTC [309] LOG: background worker "logical replication launcher" (PID 318) exited with exit code 1
2026-09-25 01:19:51.049 UTC [313] LOG: shutting down
2026-09-25 01:19:51.213 UTC [309] LOG: database system is shut down
2026-09-25T01:19:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51973 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:19:51 dist-test-slave-s12s krb5kdc[1833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:19:51 dist-test-slave-s12s krb5kdc[1833](info): set up 2 sockets
Sep 25 01:19:51 dist-test-slave-s12s krb5kdc[1833](info): commencing operation
krb5kdc: starting...
W20260925 01:19:52.449203 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:19:52 dist-test-slave-s12s krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:19:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:19:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:19:52.687861 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:42077
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:42077,127.20.102.189:42567,127.20.102.188:41349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:19:53.025897 1846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:19:53.026257 1846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:19:53.026418 1846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:19:53.033680 1846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:19:53.033821 1846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:19:53.033929 1846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:19:53.034036 1846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:19:53.034111 1846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:19:53.044912 1846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:42077,127.20.102.189:42567,127.20.102.188:41349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:42077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:19:53.046351 1846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:19:53.047701 1846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:19:53.060117 1851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:53.060280 1852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:53.060288 1854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:19:53.061164 1846 server_base.cc:1034] running on GCE node
I20260925 01:19:53.061986 1846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:19:53.063386 1846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:19:53.064639 1846 hybrid_clock.cc:648] HybridClock initialized: now 1790299193064589 us; error 43 us; skew 500 ppm
I20260925 01:19:53.065032 1846 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:19:53 dist-test-slave-s12s krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299193, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:53.072932 1846 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:19:53.074429 1846 webserver.cc:460] Webserver started at http://127.20.102.190:41157/ using document root <none> and password file <none>
I20260925 01:19:53.075219 1846 fs_manager.cc:359] Metadata directory not provided
I20260925 01:19:53.075357 1846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:19:53.075678 1846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:19:53.078743 1846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "8d8f18890a0941e0a967359aeba45f8f"
format_stamp: "Formatted at 2026-09-25 01:19:53 on dist-test-slave-s12s"
server_key: "6accda2415f4ce8cc7f059bdb4bd7796"
server_key_iv: "14d87452fb8ff2e3b39b98cd893bee1c"
server_key_version: "encryptionkey@0"
I20260925 01:19:53.079597 1846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "8d8f18890a0941e0a967359aeba45f8f"
format_stamp: "Formatted at 2026-09-25 01:19:53 on dist-test-slave-s12s"
server_key: "6accda2415f4ce8cc7f059bdb4bd7796"
server_key_iv: "14d87452fb8ff2e3b39b98cd893bee1c"
server_key_version: "encryptionkey@0"
I20260925 01:19:53.084964 1846 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 01:19:53.088379 1861 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:53.089807 1846 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:19:53.089986 1846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "8d8f18890a0941e0a967359aeba45f8f"
format_stamp: "Formatted at 2026-09-25 01:19:53 on dist-test-slave-s12s"
server_key: "6accda2415f4ce8cc7f059bdb4bd7796"
server_key_iv: "14d87452fb8ff2e3b39b98cd893bee1c"
server_key_version: "encryptionkey@0"
I20260925 01:19:53.090191 1846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:19:53.121713 1846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:19:53.125803 1846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:19:53.126080 1846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:19:53.146122 1846 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:42077
I20260925 01:19:53.146159 1912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:42077 every 8 connection(s)
I20260925 01:19:53.147547 1846 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:19:53.151144 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 1846
I20260925 01:19:53.151391 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:19:53.151662 1913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:19:53.152148 20890 external_mini_cluster.cc:1428] Setting key 40e6f00e3fdee4a6edda73979e975dbc
I20260925 01:19:53.158757 1913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.102.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.189@KRBTEST.COM" created.
Sep 25 01:19:53 dist-test-slave-s12s krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299193, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:19:53.184047 1917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.163378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52548 (local address 127.20.102.190:42077)
0925 01:19:53.164081 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:53.164091 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:53.164149 (+ 58us) server_negotiation.cc:407] Connection header received
0925 01:19:53.165630 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:53.165641 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:53.165943 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:53.166157 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:53.167397 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.169446 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.170909 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.171878 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.172093 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:53.176964 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:53.176972 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:53.176985 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:53.177015 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:53.179718 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:53.180390 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:53.180405 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:53.180407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:53.180581 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:53.181000 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:53.181006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:53.181009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:53.181514 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:53.181726 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:53.182249 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 01:19:53.182560 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":317,"threads_started":1}
I20260925 01:19:53.184085 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.161943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:42077 (local address 127.0.0.1:52548)
0925 01:19:53.162895 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.162948 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:53.165211 (+ 2263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:53.166252 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:53.166258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:53.166841 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:53.167190 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:53.167208 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.169629 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.169633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:53.170739 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:53.170752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.172109 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.172114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:53.172244 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:53.173024 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:53.173047 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:53.176748 (+ 3701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:53.179851 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:53.179864 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:53.179875 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:53.180262 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:53.180727 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:53.180732 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:53.180736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:53.180886 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:53.181647 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:53.181659 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:53.181873 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 01:19:53.182115 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 01:19:53.182701 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"mutex_wait_us":138,"thread_start_us":445,"threads_started":1}
I20260925 01:19:53.187775 1913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:19:53.188892 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.188456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42567 (local address 127.0.0.1:54426)
0925 01:19:53.188628 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.188781 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 01:19:53.189147 1862 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
I20260925 01:19:53.190876 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.190421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42567 (local address 127.0.0.1:54428)
0925 01:19:53.190657 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.190734 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260925 01:19:53.191084 1913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:42567: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.189.keytab.
Entry for principal kudu/127.20.102.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.189.keytab.
I20260925 01:19:53.204001 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.189:42567
--webserver_interface=127.20.102.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.189.keytab
--principal=kudu/127.20.102.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:42077,127.20.102.189:42567,127.20.102.188:41349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:19:53.240432 1913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } attempt: 1
I20260925 01:19:53.241715 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.241211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42567 (local address 127.0.0.1:54440)
0925 01:19:53.241439 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.241571 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 01:19:53.242703 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.242364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42567 (local address 127.0.0.1:54454)
0925 01:19:53.242502 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.242575 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 01:19:53.242954 1913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:42567: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
I20260925 01:19:53.311225 1913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } attempt: 2
I20260925 01:19:53.312600 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.312107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42567 (local address 127.0.0.1:54460)
0925 01:19:53.312360 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.312452 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 01:19:53.313735 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.313317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42567 (local address 127.0.0.1:54466)
0925 01:19:53.313488 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.313591 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 01:19:53.313931 1913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:42567: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
I20260925 01:19:53.405203 1913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } attempt: 3
I20260925 01:19:53.406394 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.405926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42567 (local address 127.0.0.1:54482)
0925 01:19:53.406139 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.406259 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 01:19:53.407387 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.407019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42567 (local address 127.0.0.1:54484)
0925 01:19:53.407170 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.407246 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 01:19:53.407586 1913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:42567: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
W20260925 01:19:53.544214 1920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:19:53.544592 1920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:19:53.544782 1920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 01:19:53.550829 1913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } attempt: 4
I20260925 01:19:53.552083 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.551496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42567 (local address 127.0.0.1:54492)
0925 01:19:53.551850 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.551938 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173}
W20260925 01:19:53.552322 1920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:19:53.552460 1920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:19:53.552577 1920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:19:53.552711 1920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:19:53.552780 1920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:19:53.553260 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.552882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42567 (local address 127.0.0.1:54500)
0925 01:19:53.553040 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.553141 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 01:19:53.553478 1913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:42567: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42567: connect: Connection refused (error 111)
I20260925 01:19:53.563067 1920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:42077,127.20.102.189:42567,127.20.102.188:41349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.189.keytab
--principal=kudu/127.20.102.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.189:42567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.102.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:19:53.564445 1920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:19:53.565817 1920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:19:53.578047 1928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:53.578081 1926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:53.578109 1925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:19:53.578799 1920 server_base.cc:1034] running on GCE node
I20260925 01:19:53.579675 1920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:19:53.581101 1920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:19:53.582340 1920 hybrid_clock.cc:648] HybridClock initialized: now 1790299193582280 us; error 51 us; skew 500 ppm
I20260925 01:19:53.582696 1920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:19:53 dist-test-slave-s12s krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299193, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:53.585944 1920 init.cc:377] Logged in from keytab as kudu/127.20.102.189@KRBTEST.COM (short username kudu)
I20260925 01:19:53.587355 1920 webserver.cc:460] Webserver started at http://127.20.102.189:42793/ using document root <none> and password file <none>
I20260925 01:19:53.588106 1920 fs_manager.cc:359] Metadata directory not provided
I20260925 01:19:53.588253 1920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:19:53.588570 1920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:19:53.591621 1920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/data/instance:
uuid: "e89b94c9603d42928ada2afa01861ffd"
format_stamp: "Formatted at 2026-09-25 01:19:53 on dist-test-slave-s12s"
server_key: "af328569cf4a175302ad51c1d482cc2f"
server_key_iv: "2fc3cf320d5128ce60e951b55b2f862a"
server_key_version: "encryptionkey@0"
I20260925 01:19:53.592412 1920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/wal/instance:
uuid: "e89b94c9603d42928ada2afa01861ffd"
format_stamp: "Formatted at 2026-09-25 01:19:53 on dist-test-slave-s12s"
server_key: "af328569cf4a175302ad51c1d482cc2f"
server_key_iv: "2fc3cf320d5128ce60e951b55b2f862a"
server_key_version: "encryptionkey@0"
I20260925 01:19:53.597712 1920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 01:19:53.601080 1935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:53.602432 1920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 01:19:53.602653 1920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/wal
uuid: "e89b94c9603d42928ada2afa01861ffd"
format_stamp: "Formatted at 2026-09-25 01:19:53 on dist-test-slave-s12s"
server_key: "af328569cf4a175302ad51c1d482cc2f"
server_key_iv: "2fc3cf320d5128ce60e951b55b2f862a"
server_key_version: "encryptionkey@0"
I20260925 01:19:53.602857 1920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:19:53.627910 1920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:19:53.631947 1920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:19:53.632266 1920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:19:53.651021 1920 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.189:42567
I20260925 01:19:53.651051 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.189:42567 every 8 connection(s)
I20260925 01:19:53.652370 1920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/data/info.pb
I20260925 01:19:53.656342 1987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:19:53.656412 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 1920
I20260925 01:19:53.656595 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-1/wal/instance
I20260925 01:19:53.657167 20890 external_mini_cluster.cc:1428] Setting key 8518af43e5603d7928877bebfea8e605
I20260925 01:19:53.666539 1987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.20.102.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.188@KRBTEST.COM" created.
Sep 25 01:19:53 dist-test-slave-s12s krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299193, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.189@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:19:53.693248 1917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.669187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52558 (local address 127.20.102.190:42077)
0925 01:19:53.669433 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:53.669441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:53.670700 (+ 1259us) server_negotiation.cc:407] Connection header received
0925 01:19:53.673712 (+ 3012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:53.673717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:53.673803 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:53.673882 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:53.676097 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.676833 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.679402 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.679834 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.679932 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:53.686641 (+ 6709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:53.686646 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:53.686649 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:53.686686 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:53.689525 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:53.690531 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:53.690535 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:53.690538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:53.690657 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:53.691133 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:53.691136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:53.691138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:53.691414 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:53.691525 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:53.692528 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0925 01:19:53.692827 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 01:19:53.694108 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.669676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:42077 (local address 127.0.0.1:52558)
0925 01:19:53.670500 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.670563 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:53.673399 (+ 2836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:53.674182 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:53.674214 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:53.675010 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:53.675868 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:53.675897 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.677020 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.677027 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:53.679245 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:53.679263 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.680075 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.680083 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:53.680390 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:53.681350 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:53.681388 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:53.686378 (+ 4990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:53.689692 (+ 3314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:53.689703 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:53.689725 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:53.690409 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:53.690770 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:53.690777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:53.690783 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:53.691024 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:53.691546 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:53.691557 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:53.692055 (+ 498us) client_negotiation.cc:769] Sending connection context
0925 01:19:53.692393 (+ 338us) client_negotiation.cc:240] Negotiation successful
0925 01:19:53.692903 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":321,"threads_started":1}
I20260925 01:19:53.695865 1987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:19:53 dist-test-slave-s12s krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299193, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.189@KRBTEST.COM for kudu/127.20.102.189@KRBTEST.COM
I20260925 01:19:53.710805 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.696475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42567 (local address 127.0.0.1:54512)
0925 01:19:53.696637 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.696659 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:53.696778 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:53.697848 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:53.697852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:53.698282 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:53.698427 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:53.698435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.699723 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.699731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:53.700585 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:53.700596 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.701427 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.701433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:53.701666 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:53.702325 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:53.702341 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:53.704558 (+ 2217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:53.708216 (+ 3658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:53.708224 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:53.708231 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:53.708752 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:53.709194 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:53.709198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:53.709200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:53.709337 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:53.710008 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:53.710011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:53.710165 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 01:19:53.710302 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 01:19:53.710501 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 01:19:53.711156 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.696778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.20.102.189:42567)
0925 01:19:53.697450 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:53.697456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:53.697493 (+ 37us) server_negotiation.cc:407] Connection header received
0925 01:19:53.697532 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:53.697537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:53.697707 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:53.697856 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:53.698580 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.699596 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.700740 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.701286 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.701423 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:53.704805 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:53.704816 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:53.704828 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:53.704861 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:53.707961 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:53.708873 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:53.708879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:53.708881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:53.709061 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:53.709446 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:53.709449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:53.709451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:53.709896 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:53.710151 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:53.710462 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 01:19:53.710739 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":372,"threads_started":1}
I20260925 01:19:53.713758 1987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.188.keytab.
Entry for principal kudu/127.20.102.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.188.keytab.
I20260925 01:19:53.715070 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.714619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:39900)
0925 01:19:53.714786 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.714929 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 01:19:53.715353 1937 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
I20260925 01:19:53.715593 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.188:41349
--webserver_interface=127.20.102.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.188.keytab
--principal=kudu/127.20.102.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:42077,127.20.102.189:42567,127.20.102.188:41349
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:19:53.717000 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.716612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:39908)
0925 01:19:53.716826 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.716903 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 01:19:53.717239 1987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:41349: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
I20260925 01:19:53.766623 1987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } attempt: 1
I20260925 01:19:53.767944 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.767365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:39910)
0925 01:19:53.767614 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.767751 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 01:19:53.769107 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.768689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:39912)
0925 01:19:53.768877 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.768962 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 01:19:53.769336 1987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:41349: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
I20260925 01:19:53.837561 1987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } attempt: 2
I20260925 01:19:53.838766 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.838246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:39928)
0925 01:19:53.838464 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.838576 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 01:19:53.839891 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.839490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:39936)
0925 01:19:53.839652 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.839733 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 01:19:53.840164 1987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:41349: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
I20260925 01:19:53.852746 1913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } attempt: 5
Sep 25 01:19:53 dist-test-slave-s12s krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299193, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for kudu/127.20.102.189@KRBTEST.COM
I20260925 01:19:53.868206 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.853555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42567 (local address 127.0.0.1:54526)
0925 01:19:53.853759 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.853788 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:53.853897 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:53.854296 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:53.854300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:53.854747 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:53.854940 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:53.854949 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.856405 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.856409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:53.857694 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:53.857711 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.858713 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.858718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:53.858875 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:53.859582 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:53.859602 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:53.861893 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:53.865855 (+ 3962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:53.865860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:53.865863 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:53.866246 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:53.866628 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:53.866632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:53.866634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:53.866788 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:53.867388 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:53.867392 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:53.867553 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 01:19:53.867688 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 01:19:53.867925 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 01:19:53.868497 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.853650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.20.102.189:42567)
0925 01:19:53.853828 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:53.853835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:53.853859 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:19:53.854033 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:53.854039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:53.854138 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:53.854258 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:53.855124 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.856235 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.857860 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:53.858553 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:53.858684 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:53.862105 (+ 3421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:53.862112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:53.862118 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:53.862162 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:53.865677 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:53.866365 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:53.866370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:53.866374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:53.866516 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:53.866896 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:53.866901 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:53.866904 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:53.867250 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:53.867393 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:53.867809 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 01:19:53.868089 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:19:53.869524 1913 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:19:53.870702 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.870288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:39944)
0925 01:19:53.870467 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.870562 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 01:19:53.871799 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.871425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:39948)
0925 01:19:53.871636 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.871707 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 01:19:53.872064 1913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:41349: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
I20260925 01:19:53.923362 1913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } attempt: 1
I20260925 01:19:53.924924 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.924162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:39958)
0925 01:19:53.924472 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.924619 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260925 01:19:53.926270 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.925771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:39964)
0925 01:19:53.925971 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.926089 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 01:19:53.926600 1913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:41349: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
I20260925 01:19:53.931404 1987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } attempt: 3
I20260925 01:19:53.932639 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.932032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:39968)
0925 01:19:53.932298 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.932424 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 01:19:53.933857 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.933398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:39984)
0925 01:19:53.933640 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.933732 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260925 01:19:53.934105 1987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:41349: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
I20260925 01:19:53.994877 1913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } attempt: 2
I20260925 01:19:53.996239 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.995624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:39986)
0925 01:19:53.995899 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.996029 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 01:19:53.997367 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:53.996938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:39998)
0925 01:19:53.997127 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:53.997231 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 01:19:53.997669 1913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:41349: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
W20260925 01:19:54.050560 1994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:19:54.050834 1994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:19:54.050997 1994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:19:54.058161 1994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:19:54.058254 1994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:19:54.058360 1994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:19:54.058470 1994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:19:54.058542 1994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:19:54.068187 1994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:42077,127.20.102.189:42567,127.20.102.188:41349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.188.keytab
--principal=kudu/127.20.102.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.188:41349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.102.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:19:54.069588 1994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:19:54.070953 1994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 01:19:54.077329 1987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } attempt: 4
I20260925 01:19:54.078524 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.078030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:40002)
0925 01:19:54.078290 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:54.078375 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 01:19:54.079614 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.079290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:40004)
0925 01:19:54.079434 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:54.079512 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 01:19:54.079922 1987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:41349: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
W20260925 01:19:54.084288 2000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:54.084254 2002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:54.084324 1999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:19:54.090376 1994 server_base.cc:1034] running on GCE node
I20260925 01:19:54.091193 1994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:19:54.092603 1994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:19:54.093813 1994 hybrid_clock.cc:648] HybridClock initialized: now 1790299194093747 us; error 53 us; skew 500 ppm
I20260925 01:19:54.094218 1994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:19:54 dist-test-slave-s12s krb5kdc[1833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299194, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:54.097514 1994 init.cc:377] Logged in from keytab as kudu/127.20.102.188@KRBTEST.COM (short username kudu)
I20260925 01:19:54.099355 1994 webserver.cc:460] Webserver started at http://127.20.102.188:44959/ using document root <none> and password file <none>
I20260925 01:19:54.100100 1994 fs_manager.cc:359] Metadata directory not provided
I20260925 01:19:54.100250 1994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:19:54.100611 1994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:19:54.103909 1913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } attempt: 3
I20260925 01:19:54.103660 1994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/data/instance:
uuid: "81e72254cb5e467e82e89bb50d1a12d6"
format_stamp: "Formatted at 2026-09-25 01:19:54 on dist-test-slave-s12s"
server_key: "7bbf62a80041f951f56ec780529df1d1"
server_key_iv: "b9b76eba62288039bb33fa91a519b366"
server_key_version: "encryptionkey@0"
I20260925 01:19:54.104610 1994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/wal/instance:
uuid: "81e72254cb5e467e82e89bb50d1a12d6"
format_stamp: "Formatted at 2026-09-25 01:19:54 on dist-test-slave-s12s"
server_key: "7bbf62a80041f951f56ec780529df1d1"
server_key_iv: "b9b76eba62288039bb33fa91a519b366"
server_key_version: "encryptionkey@0"
I20260925 01:19:54.105098 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.104510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:40006)
0925 01:19:54.104769 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:54.104884 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 01:19:54.106251 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.105766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:40016)
0925 01:19:54.106004 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:54.106103 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260925 01:19:54.106530 1913 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:41349: Network error: Client connection negotiation failed: client connection to 127.20.102.188:41349: connect: Connection refused (error 111)
I20260925 01:19:54.110540 1994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:19:54.114192 2009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:54.115705 1994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:19:54.115899 1994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/wal
uuid: "81e72254cb5e467e82e89bb50d1a12d6"
format_stamp: "Formatted at 2026-09-25 01:19:54 on dist-test-slave-s12s"
server_key: "7bbf62a80041f951f56ec780529df1d1"
server_key_iv: "b9b76eba62288039bb33fa91a519b366"
server_key_version: "encryptionkey@0"
I20260925 01:19:54.116107 1994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:19:54.129739 1994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:19:54.133915 1994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:19:54.134251 1994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:19:54.154102 1994 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.188:41349
I20260925 01:19:54.154120 2060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.188:41349 every 8 connection(s)
I20260925 01:19:54.155475 1994 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/data/info.pb
I20260925 01:19:54.158661 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 1994
I20260925 01:19:54.158859 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790298989998230-20890-0/minicluster-data/master-2/wal/instance
I20260925 01:19:54.159431 20890 external_mini_cluster.cc:1428] Setting key 519548822a6bd37bdf44edaa78b7dbfb
I20260925 01:19:54.159560 2061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:19:54.168215 2061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:19:54 dist-test-slave-s12s krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.188@KRBTEST.COM
Sep 25 01:19:54 dist-test-slave-s12s krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299194, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.188@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:19:54.182498 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.161436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40026 (local address 127.20.102.188:41349)
0925 01:19:54.162233 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:54.162245 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:54.162288 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:19:54.164364 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:54.164386 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:54.164840 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:54.165243 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:54.166740 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.167965 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.169444 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.170151 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.170333 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.175113 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:54.175121 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:54.175131 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:54.175165 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:54.178306 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.179095 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.179109 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.179118 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.179314 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.179700 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.179705 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.179709 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.180262 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:54.180497 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:54.181033 (+ 536us) server_negotiation.cc:299] Negotiation successful
0925 01:19:54.181363 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":379,"threads_started":1}
I20260925 01:19:54.186384 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.170700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:42077 (local address 127.0.0.1:52560)
0925 01:19:54.171510 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:54.171549 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:54.171725 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:54.172130 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:54.172136 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:54.172786 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:54.173045 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:54.173054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.174409 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.174417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:54.175460 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:54.175472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.176415 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.176423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:54.176663 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.177411 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:54.177428 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:54.179703 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:54.183073 (+ 3370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:54.183084 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:54.183091 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:54.183684 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:54.184109 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:54.184114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:54.184117 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:54.184273 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:54.184993 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:54.185003 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:54.185313 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 01:19:54.185536 (+ 223us) client_negotiation.cc:240] Negotiation successful
0925 01:19:54.185837 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":507,"threads_started":1}
I20260925 01:19:54.186416 1917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.170813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.20.102.190:42077)
0925 01:19:54.171037 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:54.171045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:54.171646 (+ 601us) server_negotiation.cc:407] Connection header received
0925 01:19:54.171859 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:54.171870 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:54.171932 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:54.172075 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:54.173193 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.174255 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.175613 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.176199 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.176348 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.179879 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:54.179884 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:54.179887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:54.179920 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:54.182917 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.183814 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.183820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.183823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.183991 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.184417 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.184422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.184424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.184827 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:54.185019 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:54.185741 (+ 722us) server_negotiation.cc:299] Negotiation successful
0925 01:19:54.185988 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:19:54.188238 2061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:19:54 dist-test-slave-s12s krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
Sep 25 01:19:54 dist-test-slave-s12s krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299194, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.188@KRBTEST.COM for kudu/127.20.102.189@KRBTEST.COM
I20260925 01:19:54.202095 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.188944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42567 (local address 127.0.0.1:54542)
0925 01:19:54.189099 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:54.189118 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:54.189211 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:54.189509 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:54.189513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:54.189968 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:54.190167 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:54.190180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.191137 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.191143 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:54.192303 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:54.192315 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.193130 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.193138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:54.193390 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.194016 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:54.194052 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:54.196735 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:54.199685 (+ 2950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:54.199690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:54.199695 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:54.200058 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:54.200433 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:54.200438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:54.200441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:54.200600 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:54.201121 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:54.201128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:54.201301 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 01:19:54.201475 (+ 174us) client_negotiation.cc:240] Negotiation successful
0925 01:19:54.201754 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 01:19:54.202127 1917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.185986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.20.102.190:42077)
0925 01:19:54.186698 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:54.186704 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:54.186719 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:19:54.186764 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:54.186768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:54.186842 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:54.186943 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:54.188619 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.189567 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.191138 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.191552 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.191647 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.196623 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:54.196630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:54.196635 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:54.196694 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:54.199525 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.200214 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.200218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.200220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.200340 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.200759 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.200762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.200764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.201022 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:54.201324 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:54.201548 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 01:19:54.201754 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":1045,"threads_started":1}
I20260925 01:19:54.202374 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.189048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54542 (local address 127.20.102.189:42567)
0925 01:19:54.189224 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:54.189230 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:54.189250 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:19:54.189309 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:54.189314 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:54.189387 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:54.189490 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:54.190328 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.190993 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.192467 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.192955 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.193061 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.196919 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:54.196923 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:54.196927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:54.196958 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:54.199528 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.200176 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.200182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.200186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.200319 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.200735 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.200741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.200747 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.201000 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:54.201140 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:54.201615 (+ 475us) server_negotiation.cc:299] Negotiation successful
0925 01:19:54.201826 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:19:54.203379 2061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:19:54 dist-test-slave-s12s krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299194, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.188@KRBTEST.COM for kudu/127.20.102.188@KRBTEST.COM
Sep 25 01:19:54 dist-test-slave-s12s krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.189@KRBTEST.COM
I20260925 01:19:54.216377 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.204311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40028 (local address 127.20.102.188:41349)
0925 01:19:54.204474 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:54.204479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:54.204493 (+ 14us) server_negotiation.cc:407] Connection header received
0925 01:19:54.204536 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:54.204539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:54.204592 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:54.204714 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:54.205354 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.206095 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.207272 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.207746 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.207891 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.210943 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:54.210949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:54.210954 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:54.210992 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:54.213752 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.214532 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.214536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.214538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.214648 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.215106 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.215110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.215112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.215380 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:54.215486 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:54.215945 (+ 459us) server_negotiation.cc:299] Negotiation successful
0925 01:19:54.216134 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":56}
I20260925 01:19:54.216795 2065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.204077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:40028)
0925 01:19:54.204235 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:54.204254 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:54.204372 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:54.204726 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:54.204730 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:54.205104 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:54.205223 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:54.205231 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.206209 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.206217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:54.207143 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:54.207154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.207905 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.207910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:54.208134 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.208733 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:54.208754 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:54.210779 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:54.213903 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:54.213911 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:54.213917 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:54.214409 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:54.214796 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:54.214799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:54.214801 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:54.214984 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:54.215509 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:54.215513 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:54.215683 (+ 170us) client_negotiation.cc:769] Sending connection context
0925 01:19:54.215867 (+ 184us) client_negotiation.cc:240] Negotiation successful
0925 01:19:54.216333 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":152}
I20260925 01:19:54.219527 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.204234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54554 (local address 127.20.102.189:42567)
0925 01:19:54.204415 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:54.204422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:54.204512 (+ 90us) server_negotiation.cc:407] Connection header received
0925 01:19:54.204736 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:54.204741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:54.204804 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:54.204898 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:54.206044 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.206868 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.208243 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.208660 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.208821 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.213195 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:54.213200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:54.213203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:54.213232 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:54.216398 (+ 3166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.217474 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.217478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.217480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.217631 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.218055 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.218059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.218062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.218358 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:54.218654 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:54.218961 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 01:19:54.219188 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 01:19:54.222682 2061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6: Bootstrap starting.
I20260925 01:19:54.224768 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:19:54.226614 2061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6: Neither blocks nor log segments found. Creating new log.
I20260925 01:19:54.227952 2061 log.cc:824] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6: Log is configured to *not* fsync() on all Append() calls
I20260925 01:19:54.231230 2061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6: No bootstrap required, opened a new log
I20260925 01:19:54.243276 2061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } }
I20260925 01:19:54.243762 2061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:19:54.243899 2061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e72254cb5e467e82e89bb50d1a12d6, State: Initialized, Role: FOLLOWER
I20260925 01:19:54.244776 2061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } }
I20260925 01:19:54.253150 1917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.230831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52572 (local address 127.20.102.190:42077)
0925 01:19:54.231028 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:54.231034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:54.231839 (+ 805us) server_negotiation.cc:407] Connection header received
0925 01:19:54.232149 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:54.232156 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:54.232245 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:54.232324 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:54.234936 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.236096 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.238615 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.239355 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.239481 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.243734 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:54.243744 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:54.243751 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:54.243800 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:54.248830 (+ 5030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.250522 (+ 1692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.250529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.250534 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.250724 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.251229 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.251234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.251237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.251631 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:54.251874 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:54.252453 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 01:19:54.252768 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:19:54.253880 2075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } } }
I20260925 01:19:54.254562 2075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:19:54.255921 2061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:19:54.265399 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.230748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40030 (local address 127.20.102.188:41349)
0925 01:19:54.231000 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:54.231008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:54.241953 (+ 10945us) server_negotiation.cc:407] Connection header received
0925 01:19:54.242461 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:54.242470 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:54.242571 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:54.242672 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:54.245234 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.246770 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.250030 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.251518 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.251699 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.257546 (+ 5847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:54.257559 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:54.257568 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:54.257627 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:54.261759 (+ 4132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.262711 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.262720 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.262727 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.262951 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.263609 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.263615 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.263620 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.263990 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:54.264252 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:54.264778 (+ 526us) server_negotiation.cc:299] Negotiation successful
0925 01:19:54.264997 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:19:54.266093 1992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.232080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54568 (local address 127.20.102.189:42567)
0925 01:19:54.232293 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:54.232301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:54.249222 (+ 16921us) server_negotiation.cc:407] Connection header received
0925 01:19:54.249399 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:54.249405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:54.249487 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:54.249579 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:54.251401 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.252447 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.254443 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.255151 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.255291 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.259262 (+ 3971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:54.259270 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:54.259274 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:54.259323 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:54.262896 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.263542 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.263549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.263554 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.263725 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.264213 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.264218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.264222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.264516 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:54.264788 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:54.265520 (+ 732us) server_negotiation.cc:299] Negotiation successful
0925 01:19:54.265760 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:19:54.269003 2086 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 01:19:54.271659 1994 master_runner.cc:186] Error getting master registration for 127.20.102.190:42077: OK, instance_id { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" instance_seqno: 1790299193140205 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:19:54.279325 1994 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 01:19:54.283756 1913 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } attempt: 4
Sep 25 01:19:54 dist-test-slave-s12s krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299193, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for kudu/127.20.102.188@KRBTEST.COM
I20260925 01:19:54.296926 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.284365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.20.102.188:41349)
0925 01:19:54.284538 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:54.284542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:54.284560 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:19:54.284739 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:54.284743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:54.284816 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:54.284897 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:54.285675 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.286383 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.287845 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.288406 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.288508 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.291638 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:54.291644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:54.291647 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:54.291679 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:54.294455 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.295149 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.295153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.295155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.295282 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.295694 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.295697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.295699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.295959 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:54.296064 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:54.296415 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 01:19:54.296585 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:19:54.297016 1916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.284268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:40038)
0925 01:19:54.284470 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:54.284487 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:54.284577 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:54.284937 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:54.284940 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:54.285318 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:54.285526 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:54.285535 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.286498 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.286502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:54.287685 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:54.287701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.288549 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.288553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:54.288741 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.289375 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:54.289393 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:54.291403 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:54.294643 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:54.294648 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:54.294652 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:54.295041 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:54.295420 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:54.295423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:54.295425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:54.295590 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:54.296075 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:54.296079 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:54.296213 (+ 134us) client_negotiation.cc:769] Sending connection context
0925 01:19:54.296333 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 01:19:54.296603 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 01:19:54.301646 1913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f: Bootstrap starting.
I20260925 01:19:54.304760 1913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f: Neither blocks nor log segments found. Creating new log.
I20260925 01:19:54.306294 1913 log.cc:824] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:19:54.309244 1913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f: No bootstrap required, opened a new log
I20260925 01:19:54.316860 1913 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } }
I20260925 01:19:54.317168 1913 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:19:54.317308 1913 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d8f18890a0941e0a967359aeba45f8f, State: Initialized, Role: FOLLOWER
I20260925 01:19:54.317857 1913 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } }
I20260925 01:19:54.318678 2088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } } }
I20260925 01:19:54.319020 2088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:19:54.319885 1913 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [sys.catalog]: configured and running, proceeding with master startup.
W20260925 01:19:54.329481 2099 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:19:54.329625 2099 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:19:54.328859 1846 master_runner.cc:186] Error getting master registration for 127.20.102.189:42567: OK, instance_id { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" instance_seqno: 1790299193645571 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:19:54.340113 1846 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 01:19:54.379048 1867 tablet.cc:2369] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:19:54.379151 1987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } attempt: 5
Sep 25 01:19:54 dist-test-slave-s12s krb5kdc[1833](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299193, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.189@KRBTEST.COM for kudu/127.20.102.188@KRBTEST.COM
W20260925 01:19:54.386368 2015 tablet.cc:2369] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:19:54.394634 1990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.379754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:41349 (local address 127.0.0.1:40044)
0925 01:19:54.380063 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 01:19:54.380092 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 01:19:54.380208 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:19:54.380532 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:19:54.380536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:19:54.381011 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:19:54.381178 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:54.381187 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.382194 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.382199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:54.383093 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:19:54.383104 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.383814 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.383821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:19:54.384039 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.384738 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:19:54.384765 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:19:54.387121 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:19:54.391105 (+ 3984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:54.391109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:54.391112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:54.391473 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:54.391848 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:19:54.391852 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:19:54.391854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:19:54.392007 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:19:54.392711 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:19:54.392716 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:19:54.392959 (+ 243us) client_negotiation.cc:769] Sending connection context
0925 01:19:54.393179 (+ 220us) client_negotiation.cc:240] Negotiation successful
0925 01:19:54.394229 (+ 1050us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260925 01:19:54.395395 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:54.379928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40044 (local address 127.20.102.188:41349)
0925 01:19:54.380113 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:54.380121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:54.380141 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:19:54.380327 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:54.380332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:54.380402 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:54.380506 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:54.381319 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.382048 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.383249 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:54.383656 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:54.383798 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:54.387336 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:54.387343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:54.387349 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:54.387389 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:54.390938 (+ 3549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.391584 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.391590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.391594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.391737 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:54.392108 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:54.392113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:54.392117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:54.392463 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:54.392718 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:54.394882 (+ 2164us) server_negotiation.cc:299] Negotiation successful
0925 01:19:54.395131 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:19:54.401273 1987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd: Bootstrap starting.
I20260925 01:19:54.405340 1987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd: Neither blocks nor log segments found. Creating new log.
I20260925 01:19:54.406705 1987 log.cc:824] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd: Log is configured to *not* fsync() on all Append() calls
I20260925 01:19:54.409994 1987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd: No bootstrap required, opened a new log
I20260925 01:19:54.421113 1987 raft_consensus.cc:348] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } }
I20260925 01:19:54.421591 1987 raft_consensus.cc:374] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:19:54.421727 1987 raft_consensus.cc:729] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e89b94c9603d42928ada2afa01861ffd, State: Initialized, Role: FOLLOWER
I20260925 01:19:54.422443 1987 consensus_queue.cc:260] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } }
I20260925 01:19:54.423537 2102 sys_catalog.cc:455] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } } }
I20260925 01:19:54.423877 2102 sys_catalog.cc:458] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:19:54.424861 1987 sys_catalog.cc:564] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:19:54.436393 2088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:19:54.436743 2088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } }
I20260925 01:19:54.438374 2088 leader_election.cc:288] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e89b94c9603d42928ada2afa01861ffd (127.20.102.189:42567), 81e72254cb5e467e82e89bb50d1a12d6 (127.20.102.188:41349)
I20260925 01:19:54.439419 1962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d8f18890a0941e0a967359aeba45f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e89b94c9603d42928ada2afa01861ffd" is_pre_election: true
I20260925 01:19:54.440022 1962 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d8f18890a0941e0a967359aeba45f8f in term 0.
I20260925 01:19:54.440793 2036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d8f18890a0941e0a967359aeba45f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81e72254cb5e467e82e89bb50d1a12d6" is_pre_election: true
W20260925 01:19:54.441177 2113 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:19:54.441354 2113 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:19:54.441335 2036 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d8f18890a0941e0a967359aeba45f8f in term 0.
I20260925 01:19:54.442010 1863 leader_election.cc:302] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81e72254cb5e467e82e89bb50d1a12d6, 8d8f18890a0941e0a967359aeba45f8f; no voters:
I20260925 01:19:54.442447 2088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:19:54.442617 2088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:19:54.442729 2088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:19:54.445276 1920 master_runner.cc:263] No leader master found from master e89b94c9603d42928ada2afa01861ffd
I20260925 01:19:54.445418 1920 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 01:19:54.446147 2088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } }
I20260925 01:19:54.447292 2036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d8f18890a0941e0a967359aeba45f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81e72254cb5e467e82e89bb50d1a12d6"
I20260925 01:19:54.447295 1962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d8f18890a0941e0a967359aeba45f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e89b94c9603d42928ada2afa01861ffd"
I20260925 01:19:54.447480 2036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:19:54.447513 1962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:19:54.450718 2036 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d8f18890a0941e0a967359aeba45f8f in term 1.
I20260925 01:19:54.451035 1962 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d8f18890a0941e0a967359aeba45f8f in term 1.
I20260925 01:19:54.451107 1863 leader_election.cc:302] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 81e72254cb5e467e82e89bb50d1a12d6, 8d8f18890a0941e0a967359aeba45f8f; no voters:
I20260925 01:19:54.452116 2088 leader_election.cc:288] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [CANDIDATE]: Term 1 election: Requested vote from peers e89b94c9603d42928ada2afa01861ffd (127.20.102.189:42567), 81e72254cb5e467e82e89bb50d1a12d6 (127.20.102.188:41349)
I20260925 01:19:54.452307 2088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:19:54.452445 2088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [term 1 LEADER]: Becoming Leader. State: Replica: 8d8f18890a0941e0a967359aeba45f8f, State: Running, Role: LEADER
I20260925 01:19:54.453688 2088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } }
I20260925 01:19:54.465660 2115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d8f18890a0941e0a967359aeba45f8f. Latest consensus state: current_term: 1 leader_uuid: "8d8f18890a0941e0a967359aeba45f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } } }
I20260925 01:19:54.466063 2115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [sys.catalog]: This master's current role is: LEADER
I20260925 01:19:54.466924 2117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:19:54.469377 2117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:19:54.478431 2036 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [term 1 FOLLOWER]: Refusing update from remote peer 8d8f18890a0941e0a967359aeba45f8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:19:54.478528 1962 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [term 1 FOLLOWER]: Refusing update from remote peer 8d8f18890a0941e0a967359aeba45f8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:19:54.479477 2115 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:19:54.479843 2088 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:19:54.502202 2102 sys_catalog.cc:455] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d8f18890a0941e0a967359aeba45f8f. Latest consensus state: current_term: 1 leader_uuid: "8d8f18890a0941e0a967359aeba45f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } } }
I20260925 01:19:54.502643 2102 sys_catalog.cc:458] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:19:54.504572 2118 mvcc.cc:205] Tried to move back new op lower bound from 7333065500573085696 to 7333065500491632640. Current Snapshot: MvccSnapshot[applied={T|T < 7333065500573085696}]
I20260925 01:19:54.513161 2115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d8f18890a0941e0a967359aeba45f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } } }
I20260925 01:19:54.513454 2115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [sys.catalog]: This master's current role is: LEADER
I20260925 01:19:54.515008 2075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d8f18890a0941e0a967359aeba45f8f. Latest consensus state: current_term: 1 leader_uuid: "8d8f18890a0941e0a967359aeba45f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } } }
I20260925 01:19:54.515358 2075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:19:54.518432 2088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d8f18890a0941e0a967359aeba45f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } } }
I20260925 01:19:54.519248 2088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f [sys.catalog]: This master's current role is: LEADER
I20260925 01:19:54.519598 2120 mvcc.cc:205] Tried to move back new op lower bound from 7333065500573085696 to 7333065500491632640. Current Snapshot: MvccSnapshot[applied={T|T < 7333065500573085696}]
I20260925 01:19:54.521404 2102 sys_catalog.cc:455] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d8f18890a0941e0a967359aeba45f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } } }
I20260925 01:19:54.521678 2102 sys_catalog.cc:458] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:19:54.523008 2117 catalog_manager.cc:1347] Generated new cluster ID: 4b9ea6015b094eaeb4e379725e3e0822
I20260925 01:19:54.523104 2117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:19:54.536594 2075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d8f18890a0941e0a967359aeba45f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d8f18890a0941e0a967359aeba45f8f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 42077 } } peers { permanent_uuid: "e89b94c9603d42928ada2afa01861ffd" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42567 } } peers { permanent_uuid: "81e72254cb5e467e82e89bb50d1a12d6" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 41349 } } }
I20260925 01:19:54.538388 2075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:19:54.543792 2117 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:19:54.545375 2117 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:19:54.558811 2117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d8f18890a0941e0a967359aeba45f8f: Generated new TSK 0
I20260925 01:19:54.559824 2117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:19:55.273578 2086 catalog_manager.cc:1259] Loaded cluster ID: 4b9ea6015b094eaeb4e379725e3e0822
I20260925 01:19:55.273700 2086 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6: loading cluster ID for follower catalog manager: success
I20260925 01:19:55.276243 2086 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6: acquiring CA information for follower catalog manager: success
I20260925 01:19:55.278258 2086 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 81e72254cb5e467e82e89bb50d1a12d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:19:55.443914 2113 catalog_manager.cc:1259] Loaded cluster ID: 4b9ea6015b094eaeb4e379725e3e0822
I20260925 01:19:55.444031 2113 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd: loading cluster ID for follower catalog manager: success
I20260925 01:19:55.447594 2113 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd: acquiring CA information for follower catalog manager: success
I20260925 01:19:55.449944 2113 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e89b94c9603d42928ada2afa01861ffd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:19:55.640491 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 1846
I20260925 01:19:55.641920 1908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:19:55.766129 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 1846
I20260925 01:19:55.790447 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 1920
I20260925 01:19:55.791605 1982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:19:55.898286 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 1920
I20260925 01:19:55.919262 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 1994
I20260925 01:19:55.920518 2056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:19:56.027650 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 1994
2026-09-25T01:19:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4687 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:19:56 dist-test-slave-s12s krb5kdc[2137](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:19:56 dist-test-slave-s12s krb5kdc[2137](info): set up 2 sockets
Sep 25 01:19:56 dist-test-slave-s12s krb5kdc[2137](info): commencing operation
krb5kdc: starting...
W20260925 01:19:57.141426 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:19:57 dist-test-slave-s12s krb5kdc[2137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:19:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:19:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:19:57.384500 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:41051
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:41051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:19:57.719972 2150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:19:57.720330 2150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:19:57.720511 2150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:19:57.727771 2150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:19:57.727914 2150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:19:57.728025 2150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:19:57.728132 2150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:19:57.728222 2150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:19:57.738133 2150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:41051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:41051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:19:57.739528 2150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:19:57.740904 2150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:19:57.753052 2155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:57.753150 2158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:57.753803 2156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:19:57.753846 2150 server_base.cc:1034] running on GCE node
I20260925 01:19:57.754673 2150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:19:57.756183 2150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:19:57.757408 2150 hybrid_clock.cc:648] HybridClock initialized: now 1790299197757358 us; error 41 us; skew 500 ppm
I20260925 01:19:57.757800 2150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:19:57 dist-test-slave-s12s krb5kdc[2137](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299197, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:57.760986 2150 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:19:57.762527 2150 webserver.cc:460] Webserver started at http://127.20.102.190:40375/ using document root <none> and password file <none>
I20260925 01:19:57.763175 2150 fs_manager.cc:359] Metadata directory not provided
I20260925 01:19:57.763275 2150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:19:57.763540 2150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:19:57.766480 2150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "dce716e9d42c40e8b8d07b782a830b32"
format_stamp: "Formatted at 2026-09-25 01:19:57 on dist-test-slave-s12s"
server_key: "258757d90ffcefc9a6999f4570a36334"
server_key_iv: "54ae14a47751f63a7b78d207b628fd6d"
server_key_version: "encryptionkey@0"
I20260925 01:19:57.767164 2150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "dce716e9d42c40e8b8d07b782a830b32"
format_stamp: "Formatted at 2026-09-25 01:19:57 on dist-test-slave-s12s"
server_key: "258757d90ffcefc9a6999f4570a36334"
server_key_iv: "54ae14a47751f63a7b78d207b628fd6d"
server_key_version: "encryptionkey@0"
I20260925 01:19:57.772352 2150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:19:57.775821 2165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:57.777307 2150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 01:19:57.777491 2150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "dce716e9d42c40e8b8d07b782a830b32"
format_stamp: "Formatted at 2026-09-25 01:19:57 on dist-test-slave-s12s"
server_key: "258757d90ffcefc9a6999f4570a36334"
server_key_iv: "54ae14a47751f63a7b78d207b628fd6d"
server_key_version: "encryptionkey@0"
I20260925 01:19:57.777709 2150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:19:57.794183 2150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:19:57.798348 2150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:19:57.798650 2150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:19:57.817617 2150 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:41051
I20260925 01:19:57.817680 2216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:41051 every 8 connection(s)
I20260925 01:19:57.819028 2150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:19:57.822860 2217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:19:57.827474 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 2150
I20260925 01:19:57.827689 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:19:57.828361 20890 external_mini_cluster.cc:1428] Setting key 0fad7df325d6c5e38cb3b56f5a89491e
I20260925 01:19:57.834986 2217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32: Bootstrap starting.
I20260925 01:19:57.838721 2217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32: Neither blocks nor log segments found. Creating new log.
I20260925 01:19:57.839847 2217 log.cc:824] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32: Log is configured to *not* fsync() on all Append() calls
I20260925 01:19:57.842782 2217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32: No bootstrap required, opened a new log
Sep 25 01:19:57 dist-test-slave-s12s krb5kdc[2137](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:19:57.853087 2217 raft_consensus.cc:348] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce716e9d42c40e8b8d07b782a830b32" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 41051 } }
I20260925 01:19:57.853555 2217 raft_consensus.cc:374] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:19:57.853662 2217 raft_consensus.cc:729] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dce716e9d42c40e8b8d07b782a830b32, State: Initialized, Role: FOLLOWER
I20260925 01:19:57.854234 2220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:57.830192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49122 (local address 127.20.102.190:41051)
0925 01:19:57.831085 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:57.831100 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:57.831158 (+ 58us) server_negotiation.cc:407] Connection header received
0925 01:19:57.833629 (+ 2471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:57.833657 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:57.834118 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:57.834552 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:57.836020 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:57.838088 (+ 2068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:57.839510 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:57.840731 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:57.840879 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:57.845157 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:57.845167 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:57.845185 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:57.845255 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:57.849365 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:57.850224 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:57.850232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:57.850241 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:57.850467 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:57.850887 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:57.850892 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:57.850895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:57.851687 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:57.852010 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:57.852382 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 01:19:57.852860 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":368,"threads_started":1}
I20260925 01:19:57.854415 2217 consensus_queue.cc:260] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce716e9d42c40e8b8d07b782a830b32" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 41051 } }
I20260925 01:19:57.854708 2217 raft_consensus.cc:388] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:19:57.854885 2217 raft_consensus.cc:482] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:19:57.855099 2217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:19:57.857728 2217 raft_consensus.cc:504] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce716e9d42c40e8b8d07b782a830b32" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 41051 } }
I20260925 01:19:57.858162 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:19:57.858280 2217 leader_election.cc:302] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dce716e9d42c40e8b8d07b782a830b32; no voters:
I20260925 01:19:57.858902 2217 leader_election.cc:288] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:19:57.859037 2222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:19:57.860985 2217 sys_catalog.cc:564] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:19:57.861089 2222 raft_consensus.cc:686] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [term 1 LEADER]: Becoming Leader. State: Replica: dce716e9d42c40e8b8d07b782a830b32, State: Running, Role: LEADER
I20260925 01:19:57.861760 2222 consensus_queue.cc:237] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce716e9d42c40e8b8d07b782a830b32" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 41051 } }
I20260925 01:19:57.865164 2223 sys_catalog.cc:455] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dce716e9d42c40e8b8d07b782a830b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce716e9d42c40e8b8d07b782a830b32" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 41051 } } }
I20260925 01:19:57.865149 2224 sys_catalog.cc:455] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dce716e9d42c40e8b8d07b782a830b32. Latest consensus state: current_term: 1 leader_uuid: "dce716e9d42c40e8b8d07b782a830b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dce716e9d42c40e8b8d07b782a830b32" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 41051 } } }
I20260925 01:19:57.865710 2223 sys_catalog.cc:458] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [sys.catalog]: This master's current role is: LEADER
I20260925 01:19:57.865717 2224 sys_catalog.cc:458] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32 [sys.catalog]: This master's current role is: LEADER
I20260925 01:19:57.869724 2232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:19:57.874918 2232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:19:57.884766 2232 catalog_manager.cc:1347] Generated new cluster ID: 3ab03e6e8d1e4dd388ba23ba865b2023
I20260925 01:19:57.884923 2232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:19:57.907507 2232 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:19:57.909117 2232 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:19:57.919111 2232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dce716e9d42c40e8b8d07b782a830b32: Generated new TSK 0
I20260925 01:19:57.919970 2232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:19:57.973050 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 2150
I20260925 01:19:57.974387 2212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:19:58.074527 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 2150
2026-09-25T01:19:58Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2041 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:19:58 dist-test-slave-s12s krb5kdc[2245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:19:58 dist-test-slave-s12s krb5kdc[2245](info): set up 2 sockets
Sep 25 01:19:58 dist-test-slave-s12s krb5kdc[2245](info): commencing operation
krb5kdc: starting...
W20260925 01:19:59.209259 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:19:59 dist-test-slave-s12s krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:19:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:19:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:19:59.465823 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39107
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:19:59.798080 2258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:19:59.798382 2258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:19:59.798473 2258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:19:59.806478 2258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:19:59.806603 2258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:19:59.806650 2258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:19:59.806710 2258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:19:59.806753 2258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:19:59.816854 2258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:39107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:19:59.818164 2258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:19:59.819439 2258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:19:59.831014 2264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:59.831063 2263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:19:59.831172 2266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:19:59.832171 2258 server_base.cc:1034] running on GCE node
I20260925 01:19:59.833093 2258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:19:59.834535 2258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:19:59.835780 2258 hybrid_clock.cc:648] HybridClock initialized: now 1790299199835738 us; error 47 us; skew 500 ppm
I20260925 01:19:59.836124 2258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:19:59 dist-test-slave-s12s krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299199, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:19:59.839304 2258 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:19:59.840894 2258 webserver.cc:460] Webserver started at http://127.20.102.190:38767/ using document root <none> and password file <none>
I20260925 01:19:59.841560 2258 fs_manager.cc:359] Metadata directory not provided
I20260925 01:19:59.841662 2258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:19:59.841905 2258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:19:59.844996 2258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "d75889f2ac7844a19913ab0b9c10864c"
format_stamp: "Formatted at 2026-09-25 01:19:59 on dist-test-slave-s12s"
server_key: "af7b1a735fe152b2fd731050e346cc26"
server_key_iv: "5d3c961c36bb72ec3721df003adb427b"
server_key_version: "encryptionkey@0"
I20260925 01:19:59.845721 2258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "d75889f2ac7844a19913ab0b9c10864c"
format_stamp: "Formatted at 2026-09-25 01:19:59 on dist-test-slave-s12s"
server_key: "af7b1a735fe152b2fd731050e346cc26"
server_key_iv: "5d3c961c36bb72ec3721df003adb427b"
server_key_version: "encryptionkey@0"
I20260925 01:19:59.851017 2258 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260925 01:19:59.854609 2273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:19:59.856117 2258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:19:59.856300 2258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "d75889f2ac7844a19913ab0b9c10864c"
format_stamp: "Formatted at 2026-09-25 01:19:59 on dist-test-slave-s12s"
server_key: "af7b1a735fe152b2fd731050e346cc26"
server_key_iv: "5d3c961c36bb72ec3721df003adb427b"
server_key_version: "encryptionkey@0"
I20260925 01:19:59.856491 2258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:19:59.867913 2258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:19:59.872085 2258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:19:59.872385 2258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:19:59.892118 2258 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:39107
I20260925 01:19:59.892167 2324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:39107 every 8 connection(s)
I20260925 01:19:59.893671 2258 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:19:59.897639 2325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:19:59.899521 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 2258
I20260925 01:19:59.899753 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:19:59.900452 20890 external_mini_cluster.cc:1428] Setting key 8551305975cb7898d7593a7ac96ce60c
I20260925 01:19:59.910553 2325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Bootstrap starting.
I20260925 01:19:59.914682 2325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Neither blocks nor log segments found. Creating new log.
I20260925 01:19:59.916162 2325 log.cc:824] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:19:59 dist-test-slave-s12s krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:19:59.919586 2325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: No bootstrap required, opened a new log
I20260925 01:19:59.926733 2328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:19:59.902794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.20.102.190:39107)
0925 01:19:59.903681 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0925 01:19:59.903694 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:19:59.903749 (+ 55us) server_negotiation.cc:407] Connection header received
0925 01:19:59.905554 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:19:59.905581 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:19:59.905996 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:19:59.906356 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:19:59.908253 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:59.909718 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:59.911651 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:19:59.912470 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:19:59.912759 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:19:59.918087 (+ 5328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:19:59.918097 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:19:59.918112 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:19:59.918155 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:19:59.921928 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:59.922770 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:59.922779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:59.922787 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:59.923006 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:19:59.923471 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:19:59.923484 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:19:59.923487 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:19:59.924120 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:19:59.924524 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0925 01:19:59.925074 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 01:19:59.925582 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":408,"threads_started":1}
I20260925 01:19:59.927909 2325 raft_consensus.cc:348] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } }
I20260925 01:19:59.928315 2325 raft_consensus.cc:374] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:19:59.928448 2325 raft_consensus.cc:729] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75889f2ac7844a19913ab0b9c10864c, State: Initialized, Role: FOLLOWER
I20260925 01:19:59.929093 2325 consensus_queue.cc:260] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } }
I20260925 01:19:59.929351 2325 raft_consensus.cc:388] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:19:59.929502 2325 raft_consensus.cc:482] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:19:59.929692 2325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:19:59.931998 2325 raft_consensus.cc:504] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } }
I20260925 01:19:59.932519 2325 leader_election.cc:302] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d75889f2ac7844a19913ab0b9c10864c; no voters:
I20260925 01:19:59.933197 2325 leader_election.cc:288] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:19:59.933351 2330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:19:59.934084 2330 raft_consensus.cc:686] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 1 LEADER]: Becoming Leader. State: Replica: d75889f2ac7844a19913ab0b9c10864c, State: Running, Role: LEADER
I20260925 01:19:59.934710 2330 consensus_queue.cc:237] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } }
I20260925 01:19:59.935384 2325 sys_catalog.cc:564] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:19:59.937827 2331 sys_catalog.cc:455] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d75889f2ac7844a19913ab0b9c10864c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } } }
I20260925 01:19:59.938284 2331 sys_catalog.cc:458] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: This master's current role is: LEADER
I20260925 01:19:59.938784 2332 sys_catalog.cc:455] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d75889f2ac7844a19913ab0b9c10864c. Latest consensus state: current_term: 1 leader_uuid: "d75889f2ac7844a19913ab0b9c10864c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } } }
I20260925 01:19:59.939173 2332 sys_catalog.cc:458] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: This master's current role is: LEADER
I20260925 01:19:59.939229 2337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:19:59.945262 2337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:19:59.954326 2337 catalog_manager.cc:1347] Generated new cluster ID: 245edc67f55049afa764572a83f936e6
I20260925 01:19:59.954469 2337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:19:59.969640 2337 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:19:59.971263 2337 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:19:59.981470 2337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Generated new TSK 0
I20260925 01:19:59.982645 2337 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:20:00.044943 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39107
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:20:00.393591 2351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:00.393885 2351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:00.394057 2351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:00.401805 2351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:00.401943 2351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:00.402138 2351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:20:00.412324 2351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:00.413793 2351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:00.415325 2351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:00.430096 2356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:00.430128 2359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:00.430003 2357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:00.431284 2351 server_base.cc:1034] running on GCE node
I20260925 01:20:00.431964 2351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:00.432972 2351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:00.434175 2351 hybrid_clock.cc:648] HybridClock initialized: now 1790299200434129 us; error 34 us; skew 500 ppm
I20260925 01:20:00.434526 2351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:00 dist-test-slave-s12s krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299200, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:00.437989 2351 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:20:00.439694 2351 webserver.cc:460] Webserver started at http://127.20.102.129:36791/ using document root <none> and password file <none>
I20260925 01:20:00.440426 2351 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:00.440532 2351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:00.440923 2351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:20:00.444028 2351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "b2743ab1f0ca42838749a0d149e0b0dd"
format_stamp: "Formatted at 2026-09-25 01:20:00 on dist-test-slave-s12s"
server_key: "86ad9d0cffe3187d25d17568232c381a"
server_key_iv: "96d4be974af53a8fb09a59294273edc5"
server_key_version: "encryptionkey@0"
I20260925 01:20:00.444908 2351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "b2743ab1f0ca42838749a0d149e0b0dd"
format_stamp: "Formatted at 2026-09-25 01:20:00 on dist-test-slave-s12s"
server_key: "86ad9d0cffe3187d25d17568232c381a"
server_key_iv: "96d4be974af53a8fb09a59294273edc5"
server_key_version: "encryptionkey@0"
I20260925 01:20:00.450214 2351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 01:20:00.453827 2366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:00.455333 2351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:20:00.455516 2351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "b2743ab1f0ca42838749a0d149e0b0dd"
format_stamp: "Formatted at 2026-09-25 01:20:00 on dist-test-slave-s12s"
server_key: "86ad9d0cffe3187d25d17568232c381a"
server_key_iv: "96d4be974af53a8fb09a59294273edc5"
server_key_version: "encryptionkey@0"
I20260925 01:20:00.455729 2351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:00.476130 2351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:00.480456 2351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:00.480830 2351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:00.482017 2351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:00.483373 2351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:20:00.483448 2351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:00.483536 2351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:20:00.483588 2351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:00.518393 2351 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:39795
I20260925 01:20:00.518442 2478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:39795 every 8 connection(s)
I20260925 01:20:00.519795 2351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:20:00.529755 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 2351
I20260925 01:20:00.530009 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:20:00.530884 20890 external_mini_cluster.cc:1428] Setting key ac87b726d5c932570ffb5f4209061230
Sep 25 01:20:00 dist-test-slave-s12s krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299200, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:20:00.542011 2481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:00.522506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:60341 (local address 127.20.102.190:39107)
0925 01:20:00.522965 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:00.522970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:00.524015 (+ 1045us) server_negotiation.cc:407] Connection header received
0925 01:20:00.525998 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:00.526004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:00.526096 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:00.526218 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:00.527763 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:00.528511 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:00.530223 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:00.530912 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:00.531105 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:00.535551 (+ 4446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:00.535556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:00.535560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:00.535614 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:00.538348 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:00.539327 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:00.539331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:00.539334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:00.539453 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:00.539966 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:00.539969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:00.539971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:00.540226 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:00.540338 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:00.541315 (+ 977us) server_negotiation.cc:299] Negotiation successful
0925 01:20:00.541542 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":236,"threads_started":1}
I20260925 01:20:00.543094 2482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:00.523125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:60341)
0925 01:20:00.523846 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:00.523891 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:00.525692 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:00.526471 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:00.526479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:00.526993 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:20:00.527579 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:00.527595 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:00.528713 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:00.528719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:00.530036 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:00.530048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:00.531189 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:00.531194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:00.531399 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:00.532289 (+ 890us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:20:00.532324 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:20:00.535322 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:20:00.538500 (+ 3178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:00.538512 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:00.538535 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:00.539199 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:00.539564 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:00.539572 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:00.539579 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:00.539851 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:00.540361 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:20:00.540372 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:20:00.540887 (+ 515us) client_negotiation.cc:769] Sending connection context
0925 01:20:00.541220 (+ 333us) client_negotiation.cc:240] Negotiation successful
0925 01:20:00.541784 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":268,"threads_started":1}
I20260925 01:20:00.544900 2479 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39107
I20260925 01:20:00.545261 2479 heartbeater.cc:464] Registering TS with master...
I20260925 01:20:00.546065 2479 heartbeater.cc:511] Master 127.20.102.190:39107 requested a full tablet report, sending...
I20260925 01:20:00.548360 2290 ts_manager.cc:194] Registered new tserver with Master: b2743ab1f0ca42838749a0d149e0b0dd (127.20.102.129:39795)
I20260925 01:20:00.549906 2290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:60341
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:20:00.585791 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39107
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:20:00.917549 2485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:00.917897 2485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:00.918085 2485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:00.925575 2485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:00.925738 2485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:00.925915 2485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:20:00.936268 2485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:00.937908 2485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:00.939320 2485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:00.955610 2493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:00.955731 2490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:00.955708 2491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:00.956532 2485 server_base.cc:1034] running on GCE node
I20260925 01:20:00.957237 2485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:00.958266 2485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:00.959522 2485 hybrid_clock.cc:648] HybridClock initialized: now 1790299200959471 us; error 33 us; skew 500 ppm
I20260925 01:20:00.959966 2485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:00 dist-test-slave-s12s krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299200, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:00.963462 2485 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:20:00.965368 2485 webserver.cc:460] Webserver started at http://127.20.102.130:42735/ using document root <none> and password file <none>
I20260925 01:20:00.966156 2485 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:00.966363 2485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:00.966715 2485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:20:00.970019 2485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "6084b2735503495a836de8d4e4bac29c"
format_stamp: "Formatted at 2026-09-25 01:20:00 on dist-test-slave-s12s"
server_key: "739a70249514a27589ac720de6fdd523"
server_key_iv: "3f0cc031a5d8468dcbb7ed1571573540"
server_key_version: "encryptionkey@0"
I20260925 01:20:00.970844 2485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "6084b2735503495a836de8d4e4bac29c"
format_stamp: "Formatted at 2026-09-25 01:20:00 on dist-test-slave-s12s"
server_key: "739a70249514a27589ac720de6fdd523"
server_key_iv: "3f0cc031a5d8468dcbb7ed1571573540"
server_key_version: "encryptionkey@0"
I20260925 01:20:00.976434 2485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:20:00.980335 2500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:00.981894 2485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:20:00.982100 2485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "6084b2735503495a836de8d4e4bac29c"
format_stamp: "Formatted at 2026-09-25 01:20:00 on dist-test-slave-s12s"
server_key: "739a70249514a27589ac720de6fdd523"
server_key_iv: "3f0cc031a5d8468dcbb7ed1571573540"
server_key_version: "encryptionkey@0"
I20260925 01:20:00.982323 2485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:00.997143 2485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:01.001237 2485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:01.001592 2485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:01.002617 2485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:01.003985 2485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:20:01.004099 2485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:01.004251 2485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:20:01.004330 2485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:01.038946 2485 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:41643
I20260925 01:20:01.039004 2612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:41643 every 8 connection(s)
I20260925 01:20:01.040310 2485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:20:01.049791 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 2485
I20260925 01:20:01.049994 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:20:01.050587 20890 external_mini_cluster.cc:1428] Setting key 59b05a0ebf3e885fa3865827ccd7ff09
Sep 25 01:20:01 dist-test-slave-s12s krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299200, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:20:01.064332 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:01.043258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:41527 (local address 127.20.102.190:39107)
0925 01:20:01.043750 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:01.043757 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:01.044911 (+ 1154us) server_negotiation.cc:407] Connection header received
0925 01:20:01.046734 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:01.046739 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:01.046810 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:01.046911 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:01.048407 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.049202 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.051502 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.051982 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.052123 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:01.056595 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:01.056603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:01.056606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:01.056649 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:01.060105 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:01.061374 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:01.061379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:01.061381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:01.061524 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:01.062148 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:01.062157 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:01.062163 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:01.062564 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:01.062763 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:01.063551 (+ 788us) server_negotiation.cc:299] Negotiation successful
0925 01:20:01.063843 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":220,"threads_started":1}
I20260925 01:20:01.065140 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:01.043921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:41527)
0925 01:20:01.044749 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:01.044790 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:01.046462 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:01.047136 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:01.047144 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:01.047657 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:20:01.048242 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:01.048260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.049362 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.049370 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:01.051345 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:01.051358 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.052234 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.052242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:01.052470 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:01.053246 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:20:01.053277 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:20:01.056275 (+ 2998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:20:01.060295 (+ 4020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:01.060306 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:01.060335 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:01.061226 (+ 891us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:01.061659 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:01.061668 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:01.061674 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:01.061998 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:01.062739 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:20:01.062746 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:20:01.063146 (+ 400us) client_negotiation.cc:769] Sending connection context
0925 01:20:01.063401 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 01:20:01.063855 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":305,"threads_started":1}
I20260925 01:20:01.066761 2613 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39107
I20260925 01:20:01.067126 2613 heartbeater.cc:464] Registering TS with master...
I20260925 01:20:01.067930 2613 heartbeater.cc:511] Master 127.20.102.190:39107 requested a full tablet report, sending...
I20260925 01:20:01.069516 2290 ts_manager.cc:194] Registered new tserver with Master: 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643)
I20260925 01:20:01.070411 2290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:41527
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:20:01.106228 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39107
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:20:01.446211 2619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:01.446499 2619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:01.446606 2619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:01.453850 2619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:01.453967 2619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:01.454083 2619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:20:01.464140 2619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:01.465423 2619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:01.466718 2619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:01.480178 2625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:01.480266 2627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:01.480296 2624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:01.481024 2619 server_base.cc:1034] running on GCE node
I20260925 01:20:01.481693 2619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:01.482659 2619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:01.483891 2619 hybrid_clock.cc:648] HybridClock initialized: now 1790299201483835 us; error 47 us; skew 500 ppm
I20260925 01:20:01.484223 2619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:01 dist-test-slave-s12s krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299201, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:01.487645 2619 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:20:01.489166 2619 webserver.cc:460] Webserver started at http://127.20.102.131:41625/ using document root <none> and password file <none>
I20260925 01:20:01.489905 2619 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:01.490059 2619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:01.490372 2619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:20:01.493376 2619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "c87696cdf3404ed880af0ebc60556679"
format_stamp: "Formatted at 2026-09-25 01:20:01 on dist-test-slave-s12s"
server_key: "97b7852bcaa68b4a7dc75d3b9a118ff3"
server_key_iv: "c8068911948dcb65cfbd18431ee3865b"
server_key_version: "encryptionkey@0"
I20260925 01:20:01.494235 2619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "c87696cdf3404ed880af0ebc60556679"
format_stamp: "Formatted at 2026-09-25 01:20:01 on dist-test-slave-s12s"
server_key: "97b7852bcaa68b4a7dc75d3b9a118ff3"
server_key_iv: "c8068911948dcb65cfbd18431ee3865b"
server_key_version: "encryptionkey@0"
I20260925 01:20:01.499504 2619 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:20:01.503141 2634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:01.504434 2619 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 01:20:01.504616 2619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "c87696cdf3404ed880af0ebc60556679"
format_stamp: "Formatted at 2026-09-25 01:20:01 on dist-test-slave-s12s"
server_key: "97b7852bcaa68b4a7dc75d3b9a118ff3"
server_key_iv: "c8068911948dcb65cfbd18431ee3865b"
server_key_version: "encryptionkey@0"
I20260925 01:20:01.504891 2619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:01.537019 2619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:01.542104 2619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:01.542511 2619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:01.543689 2619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:01.545161 2619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:20:01.545264 2619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:01.545423 2619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:20:01.545506 2619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:01.553332 2479 heartbeater.cc:503] Master 127.20.102.190:39107 was elected leader, sending a full tablet report...
I20260925 01:20:01.581406 2619 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:33541
I20260925 01:20:01.581449 2746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:33541 every 8 connection(s)
I20260925 01:20:01.582808 2619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:20:01.591295 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 2619
I20260925 01:20:01.591532 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:20:01.592002 20890 external_mini_cluster.cc:1428] Setting key bd9daf01e08ca16057ed7711b03ba5d9
Sep 25 01:20:01 dist-test-slave-s12s krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299201, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:20:01.603444 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:01.585479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:58503 (local address 127.20.102.190:39107)
0925 01:20:01.585962 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:01.585968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:01.587017 (+ 1049us) server_negotiation.cc:407] Connection header received
0925 01:20:01.588811 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:01.588817 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:01.588899 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:01.588997 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:01.590489 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.591304 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.593244 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.593910 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.594108 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:01.597816 (+ 3708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:01.597822 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:01.597825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:01.597859 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:01.600344 (+ 2485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:01.601191 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:01.601196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:01.601198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:01.601334 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:01.601764 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:01.601768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:01.601770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:01.602041 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:01.602173 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:01.602942 (+ 769us) server_negotiation.cc:299] Negotiation successful
0925 01:20:01.603151 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":219,"threads_started":1}
I20260925 01:20:01.604447 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:01.586107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:58503)
0925 01:20:01.586855 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:01.586898 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:01.588506 (+ 1608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:01.589207 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:01.589215 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:01.589747 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:20:01.590330 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:01.590347 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.591494 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.591498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:01.593078 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:01.593092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.594210 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.594218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:01.594472 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:01.595236 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:20:01.595272 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:20:01.597590 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:20:01.600492 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:01.600499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:01.600515 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:01.601053 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:01.601440 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:01.601443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:01.601445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:01.601658 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:01.602158 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:20:01.602163 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:20:01.602497 (+ 334us) client_negotiation.cc:769] Sending connection context
0925 01:20:01.602799 (+ 302us) client_negotiation.cc:240] Negotiation successful
0925 01:20:01.603247 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":328,"threads_started":1}
I20260925 01:20:01.606094 2747 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39107
I20260925 01:20:01.606427 2747 heartbeater.cc:464] Registering TS with master...
I20260925 01:20:01.607319 2747 heartbeater.cc:511] Master 127.20.102.190:39107 requested a full tablet report, sending...
I20260925 01:20:01.608964 2290 ts_manager.cc:194] Registered new tserver with Master: c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541)
I20260925 01:20:01.609805 2290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:58503
I20260925 01:20:01.620496 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:20:01.638646 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:01.625356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.20.102.190:39107)
0925 01:20:01.625576 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:01.625583 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:01.626044 (+ 461us) server_negotiation.cc:407] Connection header received
0925 01:20:01.626255 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:01.626260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:01.626330 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:01.626424 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:01.627843 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.628605 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.629987 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.630406 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.630547 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:01.633075 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:01.633080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:01.633084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:01.633117 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:01.635833 (+ 2716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:01.636549 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:01.636556 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:01.636560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:01.636756 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:01.637151 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:01.637157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:01.637160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:01.637500 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:01.637760 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:01.638099 (+ 339us) server_negotiation.cc:299] Negotiation successful
0925 01:20:01.638326 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:20:01.643010 2290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:20:01.649957 2290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:20:01.674741 2760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:01.663809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:33541 (local address 127.0.0.1:58568)
0925 01:20:01.665315 (+ 1506us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:01.665348 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:01.665673 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:01.666789 (+ 1116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:01.666795 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:01.666826 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:01.667187 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:01.667198 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.669247 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.669252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:01.671737 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:01.671752 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.673607 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.673613 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:01.673892 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:01.673962 (+ 70us) client_negotiation.cc:769] Sending connection context
0925 01:20:01.674099 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 01:20:01.674291 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1284,"mutex_wait_us":1,"thread_start_us":1177,"threads_started":1}
I20260925 01:20:01.675208 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:01.664047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58568 (local address 127.20.102.131:33541)
0925 01:20:01.664829 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:01.664840 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:01.665488 (+ 648us) server_negotiation.cc:407] Connection header received
0925 01:20:01.665960 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:01.665969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:01.666232 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:01.666414 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:01.667379 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.669106 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.671912 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.673410 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.673607 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:01.674596 (+ 989us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:01.674680 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 01:20:01.674878 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":416,"threads_started":1}
I20260925 01:20:01.676774 2764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:01.665829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:41643 (local address 127.0.0.1:54974)
0925 01:20:01.668920 (+ 3091us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:01.668940 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:01.669073 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:01.669925 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:01.669931 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:01.669960 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:01.670431 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:01.670443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.672195 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.672203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:01.673971 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:01.673985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.675894 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.675900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:01.676061 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:01.676093 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 01:20:01.676193 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 01:20:01.676335 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2965,"mutex_wait_us":57,"thread_start_us":2786,"threads_started":1}
I20260925 01:20:01.677292 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:01.666424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54974 (local address 127.20.102.130:41643)
0925 01:20:01.668461 (+ 2037us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:01.668470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:01.669053 (+ 583us) server_negotiation.cc:407] Connection header received
0925 01:20:01.669252 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:01.669260 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:01.669488 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:01.669673 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:01.670604 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.672053 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.674164 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.675674 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.675851 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:01.676024 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:01.676818 (+ 794us) server_negotiation.cc:299] Negotiation successful
0925 01:20:01.677015 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1891,"thread_start_us":1777,"threads_started":1}
I20260925 01:20:01.679656 2682 tablet_service.cc:1467] Processing CreateTablet for tablet 829c23a3570f4763a5c24117e493e9fb (DEFAULT_TABLE table=test-table [id=18a109931ad2412a9aa57a374cf2519b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:20:01.681316 2682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829c23a3570f4763a5c24117e493e9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:20:01.681378 2762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:01.664595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39795 (local address 127.0.0.1:45284)
0925 01:20:01.667857 (+ 3262us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:01.667878 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:01.667999 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:01.668670 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:01.668718 (+ 48us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:01.668746 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:01.669232 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:01.669246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.671218 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.671226 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:01.673220 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:01.673235 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.676195 (+ 2960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.676202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:01.676402 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:01.676435 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:20:01.680833 (+ 4398us) client_negotiation.cc:240] Negotiation successful
0925 01:20:01.680987 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3127,"mutex_wait_us":1,"thread_start_us":2983,"threads_started":1}
I20260925 01:20:01.681443 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:01.665192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45284 (local address 127.20.102.129:39795)
0925 01:20:01.665857 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:01.665866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:01.667998 (+ 2132us) server_negotiation.cc:407] Connection header received
0925 01:20:01.668142 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:01.668153 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:01.668407 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:01.669435 (+ 1028us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:01.669749 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.671080 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.673394 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:01.675939 (+ 2545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:01.676195 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:01.676358 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:01.680903 (+ 4545us) server_negotiation.cc:299] Negotiation successful
0925 01:20:01.681090 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":315,"threads_started":1}
I20260925 01:20:01.680992 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 829c23a3570f4763a5c24117e493e9fb (DEFAULT_TABLE table=test-table [id=18a109931ad2412a9aa57a374cf2519b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:20:01.682366 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829c23a3570f4763a5c24117e493e9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:20:01.684396 2414 tablet_service.cc:1467] Processing CreateTablet for tablet 829c23a3570f4763a5c24117e493e9fb (DEFAULT_TABLE table=test-table [id=18a109931ad2412a9aa57a374cf2519b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:20:01.685874 2414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 829c23a3570f4763a5c24117e493e9fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:20:01.696949 2766 tablet_bootstrap.cc:492] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd: Bootstrap starting.
I20260925 01:20:01.700443 2766 tablet_bootstrap.cc:654] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd: Neither blocks nor log segments found. Creating new log.
I20260925 01:20:01.701450 2768 tablet_bootstrap.cc:492] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679: Bootstrap starting.
I20260925 01:20:01.702386 2767 tablet_bootstrap.cc:492] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c: Bootstrap starting.
I20260925 01:20:01.703436 2766 log.cc:824] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:01.706264 2766 tablet_bootstrap.cc:492] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd: No bootstrap required, opened a new log
I20260925 01:20:01.706301 2767 tablet_bootstrap.cc:654] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c: Neither blocks nor log segments found. Creating new log.
I20260925 01:20:01.706552 2766 ts_tablet_manager.cc:1397] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 01:20:01.707548 2767 log.cc:824] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:01.707732 2768 tablet_bootstrap.cc:654] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679: Neither blocks nor log segments found. Creating new log.
I20260925 01:20:01.709131 2768 log.cc:824] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:01.711277 2767 tablet_bootstrap.cc:492] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c: No bootstrap required, opened a new log
I20260925 01:20:01.711622 2767 ts_tablet_manager.cc:1397] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 01:20:01.715008 2766 raft_consensus.cc:348] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:01.715430 2766 raft_consensus.cc:374] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:20:01.715584 2766 raft_consensus.cc:729] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2743ab1f0ca42838749a0d149e0b0dd, State: Initialized, Role: FOLLOWER
I20260925 01:20:01.716375 2766 consensus_queue.cc:260] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:01.718286 2766 ts_tablet_manager.cc:1428] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260925 01:20:01.718451 2768 tablet_bootstrap.cc:492] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679: No bootstrap required, opened a new log
I20260925 01:20:01.718806 2768 ts_tablet_manager.cc:1397] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.006s
I20260925 01:20:01.723321 2767 raft_consensus.cc:348] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:01.723842 2767 raft_consensus.cc:374] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:20:01.723958 2767 raft_consensus.cc:729] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084b2735503495a836de8d4e4bac29c, State: Initialized, Role: FOLLOWER
I20260925 01:20:01.724625 2767 consensus_queue.cc:260] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:01.725926 2613 heartbeater.cc:503] Master 127.20.102.190:39107 was elected leader, sending a full tablet report...
I20260925 01:20:01.726554 2767 ts_tablet_manager.cc:1428] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260925 01:20:01.728890 2768 raft_consensus.cc:348] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:01.729372 2768 raft_consensus.cc:374] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:20:01.729488 2768 raft_consensus.cc:729] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87696cdf3404ed880af0ebc60556679, State: Initialized, Role: FOLLOWER
I20260925 01:20:01.730091 2768 consensus_queue.cc:260] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:01.731266 2747 heartbeater.cc:503] Master 127.20.102.190:39107 was elected leader, sending a full tablet report...
I20260925 01:20:01.731791 2768 ts_tablet_manager.cc:1428] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
W20260925 01:20:01.772363 2480 tablet.cc:2369] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:20:01.792570 2614 tablet.cc:2369] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:20:01.834537 2748 tablet.cc:2369] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:20:02.001583 2774 raft_consensus.cc:482] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:20:02.001916 2774 raft_consensus.cc:504] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:02.003830 2774 leader_election.cc:288] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643), b2743ab1f0ca42838749a0d149e0b0dd (127.20.102.129:39795)
I20260925 01:20:02.011441 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:02.004423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:46661 (local address 127.20.102.130:41643)
0925 01:20:02.004727 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:02.004737 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:02.004765 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:20:02.004854 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:02.004861 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:02.004950 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:02.005113 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:02.005512 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.006687 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.009243 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.010514 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.010689 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:02.010774 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:02.011046 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 01:20:02.011179 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 01:20:02.011466 2750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:02.004071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:41643 (local address 127.20.102.131:46661)
0925 01:20:02.004340 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:02.004364 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:02.004504 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:02.005134 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:02.005139 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:02.005162 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:02.005370 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:02.005379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.006821 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.006825 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:02.009031 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:02.009045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.010658 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.010663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:02.010850 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:02.010899 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 01:20:02.010996 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 01:20:02.011137 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":91}
I20260925 01:20:02.011695 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:02.004485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:58529 (local address 127.20.102.129:39795)
0925 01:20:02.004751 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:02.004759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:02.005119 (+ 360us) server_negotiation.cc:407] Connection header received
0925 01:20:02.005294 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:02.005300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:02.005378 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:02.005495 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:02.005943 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.007032 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.009452 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.010715 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.010868 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:02.010917 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:02.011227 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 01:20:02.011421 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:20:02.012184 2775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:02.004315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39795 (local address 127.20.102.131:58529)
0925 01:20:02.004982 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:02.004995 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:02.005134 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:02.005495 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:02.005500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:02.005529 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:02.005781 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:02.005789 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.007174 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.007180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:02.009299 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:02.009313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.010905 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.010910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:02.011070 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:02.011099 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 01:20:02.011177 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 01:20:02.011307 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":434,"threads_started":1}
I20260925 01:20:02.012737 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829c23a3570f4763a5c24117e493e9fb" candidate_uuid: "c87696cdf3404ed880af0ebc60556679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084b2735503495a836de8d4e4bac29c" is_pre_election: true
I20260925 01:20:02.013402 2568 raft_consensus.cc:2454] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87696cdf3404ed880af0ebc60556679 in term 0.
I20260925 01:20:02.013286 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829c23a3570f4763a5c24117e493e9fb" candidate_uuid: "c87696cdf3404ed880af0ebc60556679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" is_pre_election: true
I20260925 01:20:02.013880 2434 raft_consensus.cc:2454] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87696cdf3404ed880af0ebc60556679 in term 0.
I20260925 01:20:02.014154 2638 leader_election.cc:302] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084b2735503495a836de8d4e4bac29c, c87696cdf3404ed880af0ebc60556679; no voters:
I20260925 01:20:02.014549 2774 raft_consensus.cc:2781] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:20:02.014715 2774 raft_consensus.cc:482] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:20:02.014829 2774 raft_consensus.cc:3037] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:20:02.017047 2774 raft_consensus.cc:504] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:02.017642 2774 leader_election.cc:288] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [CANDIDATE]: Term 1 election: Requested vote from peers 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643), b2743ab1f0ca42838749a0d149e0b0dd (127.20.102.129:39795)
I20260925 01:20:02.018072 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829c23a3570f4763a5c24117e493e9fb" candidate_uuid: "c87696cdf3404ed880af0ebc60556679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6084b2735503495a836de8d4e4bac29c"
I20260925 01:20:02.018072 2434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "829c23a3570f4763a5c24117e493e9fb" candidate_uuid: "c87696cdf3404ed880af0ebc60556679" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2743ab1f0ca42838749a0d149e0b0dd"
I20260925 01:20:02.018287 2434 raft_consensus.cc:3037] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:20:02.018286 2568 raft_consensus.cc:3037] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:20:02.021610 2568 raft_consensus.cc:2454] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87696cdf3404ed880af0ebc60556679 in term 1.
I20260925 01:20:02.021601 2434 raft_consensus.cc:2454] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87696cdf3404ed880af0ebc60556679 in term 1.
I20260925 01:20:02.022037 2636 leader_election.cc:302] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2743ab1f0ca42838749a0d149e0b0dd, c87696cdf3404ed880af0ebc60556679; no voters:
I20260925 01:20:02.022397 2774 raft_consensus.cc:2781] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:20:02.022905 2774 raft_consensus.cc:686] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [term 1 LEADER]: Becoming Leader. State: Replica: c87696cdf3404ed880af0ebc60556679, State: Running, Role: LEADER
I20260925 01:20:02.023478 2774 consensus_queue.cc:237] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:02.028353 2290 catalog_manager.cc:5495] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 reported cstate change: term changed from 0 to 1, leader changed from <none> to c87696cdf3404ed880af0ebc60556679 (127.20.102.131). New cstate: current_term: 1 leader_uuid: "c87696cdf3404ed880af0ebc60556679" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } health_report { overall_health: UNKNOWN } } }
I20260925 01:20:02.129748 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:02.124549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.20.102.131:33541)
0925 01:20:02.124801 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:02.124805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:02.124823 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:20:02.124952 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:02.124955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:02.125034 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:02.125168 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:20:02.125512 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.126260 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.127605 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.127991 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.128080 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:02.128609 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:20:02.129263 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:20:02.129384 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:02.129472 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 01:20:02.129539 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:20:02.138249 2434 raft_consensus.cc:1261] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Refusing update from remote peer c87696cdf3404ed880af0ebc60556679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:20:02.138249 2568 raft_consensus.cc:1261] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c [term 1 FOLLOWER]: Refusing update from remote peer c87696cdf3404ed880af0ebc60556679: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:20:02.139307 2776 consensus_queue.cc:1035] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:20:02.139668 2774 consensus_queue.cc:1035] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:20:02.152958 2779 mvcc.cc:205] Tried to move back new op lower bound from 7333065531947503616 to 7333065531492315136. Current Snapshot: MvccSnapshot[applied={T|T < 7333065531947503616}]
I20260925 01:20:02.166087 2289 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:20:02.166374 2289 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:20:02.169931 2289 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Sending DeleteTablet for 3 replicas of tablet 829c23a3570f4763a5c24117e493e9fb
I20260925 01:20:02.171235 2414 tablet_service.cc:1514] Processing DeleteTablet for tablet 829c23a3570f4763a5c24117e493e9fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:20:02 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:45284
I20260925 01:20:02.171375 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 2351
I20260925 01:20:02.171483 2682 tablet_service.cc:1514] Processing DeleteTablet for tablet 829c23a3570f4763a5c24117e493e9fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:20:02 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:58568
I20260925 01:20:02.171991 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet 829c23a3570f4763a5c24117e493e9fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:20:02 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:54974
I20260925 01:20:02.181960 2792 tablet_replica.cc:330] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c: stopping tablet replica
I20260925 01:20:02.182572 2792 raft_consensus.cc:2229] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:20:02.183310 2792 raft_consensus.cc:2258] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:20:02.186827 2789 tablet_replica.cc:330] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd: stopping tablet replica
I20260925 01:20:02.187500 2789 raft_consensus.cc:2229] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:20:02.188201 2789 raft_consensus.cc:2258] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:20:02.190721 2792 ts_tablet_manager.cc:1905] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:20:02.190721 2789 ts_tablet_manager.cc:1905] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:20:02.198025 2789 ts_tablet_manager.cc:1918] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:20:02.198323 2789 log.cc:1196] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/829c23a3570f4763a5c24117e493e9fb
I20260925 01:20:02.198971 2789 ts_tablet_manager.cc:1939] T 829c23a3570f4763a5c24117e493e9fb P b2743ab1f0ca42838749a0d149e0b0dd: Deleting consensus metadata
I20260925 01:20:02.199671 2790 tablet_replica.cc:330] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679: stopping tablet replica
I20260925 01:20:02.201193 2275 catalog_manager.cc:4841] TS b2743ab1f0ca42838749a0d149e0b0dd (127.20.102.129:39795): tablet 829c23a3570f4763a5c24117e493e9fb (table test-table [id=18a109931ad2412a9aa57a374cf2519b]) successfully deleted
I20260925 01:20:02.202095 2790 raft_consensus.cc:2229] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:20:02.202242 2792 ts_tablet_manager.cc:1918] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:20:02.202425 2792 log.cc:1196] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/829c23a3570f4763a5c24117e493e9fb
I20260925 01:20:02.202812 2790 raft_consensus.cc:2258] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:20:02.203027 2792 ts_tablet_manager.cc:1939] T 829c23a3570f4763a5c24117e493e9fb P 6084b2735503495a836de8d4e4bac29c: Deleting consensus metadata
I20260925 01:20:02.204977 2277 catalog_manager.cc:4841] TS 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643): tablet 829c23a3570f4763a5c24117e493e9fb (table test-table [id=18a109931ad2412a9aa57a374cf2519b]) successfully deleted
I20260925 01:20:02.206441 2790 ts_tablet_manager.cc:1905] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 01:20:02 dist-test-slave-s12s krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:20:02.214113 2763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:02.172570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36394 (local address 127.20.102.129:39795)
0925 01:20:02.172796 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:02.172804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:02.186305 (+ 13501us) server_negotiation.cc:407] Connection header received
0925 01:20:02.187040 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:02.187048 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:02.187136 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:02.187218 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:02.195690 (+ 8472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.196920 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.199886 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.200516 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.200626 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:02.207799 (+ 7173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:02.207813 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:02.207826 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:02.207862 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:02.210809 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:02.211645 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:02.211653 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:02.211656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:02.211790 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:02.212256 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:02.212259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:02.212261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:02.212626 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:02.213037 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:02.213394 (+ 357us) server_negotiation.cc:299] Negotiation successful
0925 01:20:02.213636 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:20:02.214099 2790 ts_tablet_manager.cc:1918] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:20:02.214263 2790 log.cc:1196] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/829c23a3570f4763a5c24117e493e9fb
I20260925 01:20:02.214872 2474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:02.214893 2790 ts_tablet_manager.cc:1939] T 829c23a3570f4763a5c24117e493e9fb P c87696cdf3404ed880af0ebc60556679: Deleting consensus metadata
I20260925 01:20:02.216735 2277 catalog_manager.cc:4841] TS c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541): tablet 829c23a3570f4763a5c24117e493e9fb (table test-table [id=18a109931ad2412a9aa57a374cf2519b]) successfully deleted
I20260925 01:20:02.365044 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 2351
I20260925 01:20:02.391948 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 2485
Sep 25 01:20:02 dist-test-slave-s12s krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:20:02.410852 2765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:02.392847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.20.102.130:41643)
0925 01:20:02.393023 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:02.393027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:02.393085 (+ 58us) server_negotiation.cc:407] Connection header received
0925 01:20:02.393291 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:02.393295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:02.393361 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:02.393439 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:02.395155 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.396051 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.398238 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.398740 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.398853 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:02.404547 (+ 5694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:02.404566 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:02.404583 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:02.404667 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:02.407679 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:02.408602 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:02.408612 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:02.408615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:02.408804 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:02.409264 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:02.409267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:02.409269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:02.409721 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:02.409938 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:02.410371 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 01:20:02.410539 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:20:02.411731 2608 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:02.502677 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 2485
I20260925 01:20:02.521793 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 2619
Sep 25 01:20:02 dist-test-slave-s12s krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299199, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:20:02.542908 2761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:02.522653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37352 (local address 127.20.102.131:33541)
0925 01:20:02.522886 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:02.522894 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:02.523199 (+ 305us) server_negotiation.cc:407] Connection header received
0925 01:20:02.523390 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:02.523394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:02.523459 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:02.523553 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:02.525428 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.526239 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.528309 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:02.528902 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:02.529022 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:02.535018 (+ 5996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:02.535032 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:02.535050 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:02.535096 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:02.539102 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:02.540021 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:02.540031 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:02.540034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:02.540246 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:02.540846 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:02.540852 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:02.540855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:02.541346 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:02.541696 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:02.542081 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 01:20:02.542350 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:20:02.543893 2742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:02.669162 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 2619
I20260925 01:20:02.690753 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 2258
I20260925 01:20:02.692152 2320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:02.804030 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 2258
I20260925 01:20:02.826563 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39107
--webserver_interface=127.20.102.190
--webserver_port=38767
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:20:03.175865 2797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:03.176168 2797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:03.176321 2797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:03.183851 2797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:20:03.183956 2797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:03.184054 2797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:03.184154 2797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:20:03.184231 2797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:20:03.194343 2797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:39107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=38767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:03.195729 2797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:03.197183 2797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:03.221839 2803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:03.221905 2806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:03.221872 2804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:03.222931 2797 server_base.cc:1034] running on GCE node
I20260925 01:20:03.223930 2797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:03.225771 2797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:03.226991 2797 hybrid_clock.cc:648] HybridClock initialized: now 1790299203226944 us; error 38 us; skew 500 ppm
I20260925 01:20:03.227440 2797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:03 dist-test-slave-s12s krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299203, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:03.238219 2797 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:20:03.240199 2797 webserver.cc:460] Webserver started at http://127.20.102.190:38767/ using document root <none> and password file <none>
I20260925 01:20:03.241137 2797 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:03.241317 2797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:03.248394 2797 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 01:20:03.251766 2813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:03.253937 2797 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:20:03.254172 2797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "d75889f2ac7844a19913ab0b9c10864c"
format_stamp: "Formatted at 2026-09-25 01:19:59 on dist-test-slave-s12s"
server_key: "af7b1a735fe152b2fd731050e346cc26"
server_key_iv: "5d3c961c36bb72ec3721df003adb427b"
server_key_version: "encryptionkey@0"
I20260925 01:20:03.255676 2797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:03.275534 2797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:03.280483 2797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:03.280848 2797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:03.300951 2797 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:39107
I20260925 01:20:03.300971 2864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:39107 every 8 connection(s)
I20260925 01:20:03.302636 2797 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:20:03.310091 2865 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:20:03.311169 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 2797
I20260925 01:20:03.311998 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:39795
--local_ip_for_outbound_sockets=127.20.102.129
--tserver_master_addrs=127.20.102.190:39107
--webserver_port=36791
--webserver_interface=127.20.102.129
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:20:03.313975 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Bootstrap starting.
I20260925 01:20:03.334712 2865 log.cc:824] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:03.347301 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:20:03.347886 2865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Bootstrap complete.
I20260925 01:20:03.357602 2865 raft_consensus.cc:348] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } }
I20260925 01:20:03.358629 2865 raft_consensus.cc:729] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75889f2ac7844a19913ab0b9c10864c, State: Initialized, Role: FOLLOWER
I20260925 01:20:03.359368 2865 consensus_queue.cc:260] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } }
I20260925 01:20:03.359670 2865 raft_consensus.cc:388] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:20:03.359817 2865 raft_consensus.cc:482] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:20:03.360049 2865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:20:03.364251 2865 raft_consensus.cc:504] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } }
I20260925 01:20:03.364916 2865 leader_election.cc:302] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d75889f2ac7844a19913ab0b9c10864c; no voters:
I20260925 01:20:03.365592 2865 leader_election.cc:288] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 01:20:03.365748 2869 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:20:03.366935 2869 raft_consensus.cc:686] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 2 LEADER]: Becoming Leader. State: Replica: d75889f2ac7844a19913ab0b9c10864c, State: Running, Role: LEADER
I20260925 01:20:03.367658 2869 consensus_queue.cc:237] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } }
I20260925 01:20:03.368117 2865 sys_catalog.cc:564] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:20:03.370232 2871 sys_catalog.cc:455] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d75889f2ac7844a19913ab0b9c10864c. Latest consensus state: current_term: 2 leader_uuid: "d75889f2ac7844a19913ab0b9c10864c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } } }
I20260925 01:20:03.370620 2871 sys_catalog.cc:458] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: This master's current role is: LEADER
I20260925 01:20:03.370410 2870 sys_catalog.cc:455] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d75889f2ac7844a19913ab0b9c10864c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } } }
I20260925 01:20:03.371083 2870 sys_catalog.cc:458] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: This master's current role is: LEADER
I20260925 01:20:03.372133 2876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:20:03.391291 2876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:20:03.392983 2876 catalog_manager.cc:1259] Loaded cluster ID: 245edc67f55049afa764572a83f936e6
I20260925 01:20:03.393184 2876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:20:03.396602 2876 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 01:20:03.415820 2876 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 01:20:03.435390 2876 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:20:03.437417 2876 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Loaded TSK: 0
I20260925 01:20:03.439157 2876 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 01:20:03.685199 2867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:03.685484 2867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:03.685672 2867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:03.693143 2867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:03.693259 2867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:03.693428 2867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:20:03.703596 2867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:39795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=36791
--tserver_master_addrs=127.20.102.190:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:03.705019 2867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:03.706370 2867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:03.720387 2895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:03.720417 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:03.720719 2892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:03.721231 2867 server_base.cc:1034] running on GCE node
I20260925 01:20:03.721873 2867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:03.723014 2867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:03.724309 2867 hybrid_clock.cc:648] HybridClock initialized: now 1790299203724231 us; error 60 us; skew 500 ppm
I20260925 01:20:03.724815 2867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:03 dist-test-slave-s12s krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299203, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:03.728477 2867 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:20:03.730427 2867 webserver.cc:460] Webserver started at http://127.20.102.129:36791/ using document root <none> and password file <none>
I20260925 01:20:03.731228 2867 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:03.731408 2867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:03.737573 2867 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 01:20:03.740792 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:03.742442 2867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:20:03.742631 2867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "b2743ab1f0ca42838749a0d149e0b0dd"
format_stamp: "Formatted at 2026-09-25 01:20:00 on dist-test-slave-s12s"
server_key: "86ad9d0cffe3187d25d17568232c381a"
server_key_iv: "96d4be974af53a8fb09a59294273edc5"
server_key_version: "encryptionkey@0"
I20260925 01:20:03.743741 2867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:03.756665 2867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:03.761713 2867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:03.762014 2867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:03.763238 2867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:03.764953 2867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:20:03.765074 2867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:03.765209 2867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:20:03.765314 2867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:03.802103 2867 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:39795
I20260925 01:20:03.802173 3014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:39795 every 8 connection(s)
I20260925 01:20:03.803766 2867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:20:03.807338 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 2867
I20260925 01:20:03.808199 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:41643
--local_ip_for_outbound_sockets=127.20.102.130
--tserver_master_addrs=127.20.102.190:39107
--webserver_port=42735
--webserver_interface=127.20.102.130
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:20:03 dist-test-slave-s12s krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299203, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:20:03.832818 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:03.806907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:40261 (local address 127.20.102.190:39107)
0925 01:20:03.807743 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:03.807761 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:03.808594 (+ 833us) server_negotiation.cc:407] Connection header received
0925 01:20:03.812030 (+ 3436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:03.812057 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:03.812479 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:03.812857 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:03.815112 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:03.816946 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:03.819000 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:03.819759 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:03.819928 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:03.824406 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:03.824415 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:03.824432 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:03.824472 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:03.827561 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:03.828826 (+ 1265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:03.828834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:03.828841 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:03.829018 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:03.829617 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:03.829621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:03.829623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:03.830167 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:03.830322 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:03.831418 (+ 1096us) server_negotiation.cc:299] Negotiation successful
0925 01:20:03.831760 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":329,"threads_started":1}
I20260925 01:20:03.833407 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:03.806892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:40261)
0925 01:20:03.808148 (+ 1256us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:03.808216 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:03.811330 (+ 3114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:03.813063 (+ 1733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:03.813078 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:03.813980 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:20:03.814852 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:03.814890 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:03.817129 (+ 2239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:03.817133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:03.818832 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:03.818850 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:03.820014 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:03.820023 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:03.820364 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:03.821175 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:20:03.821211 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:20:03.824107 (+ 2896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:20:03.827786 (+ 3679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:03.827801 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:03.827822 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:03.828587 (+ 765us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:03.829174 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:03.829182 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:03.829188 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:03.829480 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:03.830346 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:20:03.830357 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:20:03.830875 (+ 518us) client_negotiation.cc:769] Sending connection context
0925 01:20:03.831251 (+ 376us) client_negotiation.cc:240] Negotiation successful
0925 01:20:03.831906 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"thread_start_us":708,"threads_started":1}
I20260925 01:20:03.836402 3015 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39107
I20260925 01:20:03.836823 3015 heartbeater.cc:464] Registering TS with master...
I20260925 01:20:03.837543 3015 heartbeater.cc:511] Master 127.20.102.190:39107 requested a full tablet report, sending...
I20260925 01:20:03.839838 2830 ts_manager.cc:194] Registered new tserver with Master: b2743ab1f0ca42838749a0d149e0b0dd (127.20.102.129:39795)
I20260925 01:20:03.841432 2830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:40261
W20260925 01:20:04.151527 3019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:04.151911 3019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:04.152089 3019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:04.159835 3019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:04.160000 3019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:04.160197 3019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:20:04.170871 3019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:41643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=42735
--tserver_master_addrs=127.20.102.190:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:04.172371 3019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:04.173933 3019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:04.187800 3024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:04.187882 3025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:04.188089 3027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:04.188987 3019 server_base.cc:1034] running on GCE node
I20260925 01:20:04.189620 3019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:04.190620 3019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:04.191897 3019 hybrid_clock.cc:648] HybridClock initialized: now 1790299204191849 us; error 36 us; skew 500 ppm
I20260925 01:20:04.192277 3019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:04 dist-test-slave-s12s krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299204, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:04.195770 3019 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:20:04.197333 3019 webserver.cc:460] Webserver started at http://127.20.102.130:42735/ using document root <none> and password file <none>
I20260925 01:20:04.198015 3019 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:04.198143 3019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:04.203956 3019 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 01:20:04.206750 3034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:04.208104 3019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:20:04.208241 3019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "6084b2735503495a836de8d4e4bac29c"
format_stamp: "Formatted at 2026-09-25 01:20:00 on dist-test-slave-s12s"
server_key: "739a70249514a27589ac720de6fdd523"
server_key_iv: "3f0cc031a5d8468dcbb7ed1571573540"
server_key_version: "encryptionkey@0"
I20260925 01:20:04.209383 3019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:04.226517 3019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:04.230990 3019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:04.231225 3019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:04.232196 3019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:04.233574 3019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:20:04.233662 3019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:04.233745 3019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:20:04.233808 3019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:04.267314 3019 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:41643
I20260925 01:20:04.267395 3146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:41643 every 8 connection(s)
I20260925 01:20:04.269215 3019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:20:04.276445 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 3019
I20260925 01:20:04.277280 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:33541
--local_ip_for_outbound_sockets=127.20.102.131
--tserver_master_addrs=127.20.102.190:39107
--webserver_port=41625
--webserver_interface=127.20.102.131
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:20:04 dist-test-slave-s12s krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299204, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:20:04.293967 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.271895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:52411 (local address 127.20.102.190:39107)
0925 01:20:04.272121 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:04.272127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:04.273498 (+ 1371us) server_negotiation.cc:407] Connection header received
0925 01:20:04.275793 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:04.275797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:04.275870 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:04.275963 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:04.278222 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.278994 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.281650 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.282103 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.282211 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.286822 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:04.286828 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:04.286832 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:04.286867 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:04.290399 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:04.291198 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:04.291207 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:04.291214 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:04.291377 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:04.291821 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:04.291829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:04.291835 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:04.292229 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:04.292390 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:04.293198 (+ 808us) server_negotiation.cc:299] Negotiation successful
0925 01:20:04.293493 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:20:04.294644 3149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.272513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:52411)
0925 01:20:04.273306 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:04.273352 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:04.275512 (+ 2160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:04.276281 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:04.276293 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:04.277100 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:20:04.277994 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:04.278023 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.279190 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.279199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:04.281466 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:04.281486 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.282357 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.282373 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:04.282731 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.283515 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:20:04.283551 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:20:04.286540 (+ 2989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:20:04.290572 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:04.290592 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:04.290607 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:04.291045 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:04.291489 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:04.291492 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:04.291494 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:04.291700 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:04.292351 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:20:04.292356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:20:04.292800 (+ 444us) client_negotiation.cc:769] Sending connection context
0925 01:20:04.293033 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 01:20:04.293445 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":348,"threads_started":1}
I20260925 01:20:04.296271 3147 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39107
I20260925 01:20:04.296605 3147 heartbeater.cc:464] Registering TS with master...
I20260925 01:20:04.297401 3147 heartbeater.cc:511] Master 127.20.102.190:39107 requested a full tablet report, sending...
I20260925 01:20:04.299008 2830 ts_manager.cc:194] Registered new tserver with Master: 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643)
I20260925 01:20:04.300012 2830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:52411
W20260925 01:20:04.629256 3150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:04.629732 3150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:04.629887 3150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:04.637401 3150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:04.637517 3150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:04.637707 3150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:20:04.647637 3150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:33541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=41625
--tserver_master_addrs=127.20.102.190:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:04.649025 3150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:04.650326 3150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:04.663900 3156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:04.663967 3158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:04.663971 3155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:04.665189 3150 server_base.cc:1034] running on GCE node
I20260925 01:20:04.665786 3150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:04.666651 3150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:04.667851 3150 hybrid_clock.cc:648] HybridClock initialized: now 1790299204667798 us; error 40 us; skew 500 ppm
I20260925 01:20:04.668222 3150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:04 dist-test-slave-s12s krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299204, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:04.671463 3150 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:20:04.673182 3150 webserver.cc:460] Webserver started at http://127.20.102.131:41625/ using document root <none> and password file <none>
I20260925 01:20:04.673995 3150 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:04.674208 3150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:04.679883 3150 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:20:04.682739 3165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:04.684206 3150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:20:04.684405 3150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "c87696cdf3404ed880af0ebc60556679"
format_stamp: "Formatted at 2026-09-25 01:20:01 on dist-test-slave-s12s"
server_key: "97b7852bcaa68b4a7dc75d3b9a118ff3"
server_key_iv: "c8068911948dcb65cfbd18431ee3865b"
server_key_version: "encryptionkey@0"
I20260925 01:20:04.685477 3150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:04.708190 3150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:04.712932 3150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:04.713217 3150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:04.714296 3150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:04.715636 3150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:20:04.715759 3150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:04.715893 3150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:20:04.715984 3150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:04.749801 3150 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:33541
I20260925 01:20:04.749890 3277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:33541 every 8 connection(s)
I20260925 01:20:04.751284 3150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:20:04.756440 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 3150
Sep 25 01:20:04 dist-test-slave-s12s krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299204, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:20:04.772859 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.757410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.20.102.190:39107)
0925 01:20:04.757909 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:04.757912 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:04.758372 (+ 460us) server_negotiation.cc:407] Connection header received
0925 01:20:04.758568 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:04.758573 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:04.758637 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:04.758743 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:04.760720 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.761553 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.763325 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.763963 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.764105 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.767021 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:04.767025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:04.767029 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:04.767067 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:04.770140 (+ 3073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:04.770818 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:04.770825 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:04.770828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:04.770955 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:04.771326 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:04.771332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:04.771335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:04.771615 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:04.771872 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:04.772171 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 01:20:04.772411 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":294,"threads_started":1}
I20260925 01:20:04.772979 3017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.753978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:44935 (local address 127.20.102.190:39107)
0925 01:20:04.754167 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:04.754171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:04.755246 (+ 1075us) server_negotiation.cc:407] Connection header received
0925 01:20:04.757208 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:04.757214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:04.757308 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:04.757414 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:04.759243 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.760478 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.762179 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.762819 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.762954 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.766381 (+ 3427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:04.766389 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:04.766394 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:04.766434 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:04.769760 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:04.770451 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:04.770456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:04.770460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:04.770588 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:04.771068 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:04.771074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:04.771077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:04.771344 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:04.771514 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:04.772211 (+ 697us) server_negotiation.cc:299] Negotiation successful
0925 01:20:04.772582 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":139,"server-negotiator.queue_time_us":69}
I20260925 01:20:04.773972 3280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.754365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:44935)
0925 01:20:04.755068 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:04.755111 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:04.756942 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:04.757746 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:04.757763 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:04.758475 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:20:04.759046 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:04.759093 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.760667 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.760671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:04.762041 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:04.762053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.763054 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.763059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:04.763257 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.763952 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:20:04.763980 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:20:04.766160 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:20:04.769896 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:04.769904 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:04.769920 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:04.770326 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:04.770698 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:04.770703 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:04.770707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:04.770954 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:04.771485 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:20:04.771493 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:20:04.771839 (+ 346us) client_negotiation.cc:769] Sending connection context
0925 01:20:04.772113 (+ 274us) client_negotiation.cc:240] Negotiation successful
0925 01:20:04.772646 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":275,"threads_started":1}
I20260925 01:20:04.775489 3278 heartbeater.cc:347] Connected to a master server at 127.20.102.190:39107
I20260925 01:20:04.775804 3278 heartbeater.cc:464] Registering TS with master...
I20260925 01:20:04.776538 3278 heartbeater.cc:511] Master 127.20.102.190:39107 requested a full tablet report, sending...
I20260925 01:20:04.777987 2830 ts_manager.cc:194] Registered new tserver with Master: c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541)
I20260925 01:20:04.778774 2830 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:44935
I20260925 01:20:04.788022 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:20:04.795187 3281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.789021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36046 (local address 127.20.102.190:39107)
0925 01:20:04.789292 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:04.789299 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:04.789636 (+ 337us) server_negotiation.cc:407] Connection header received
0925 01:20:04.789896 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:04.789900 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:04.790094 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:04.790225 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:20:04.790671 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.791362 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.792904 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.793358 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.793464 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.793980 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:20:04.794573 (+ 593us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:20:04.794689 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:04.794805 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 01:20:04.794875 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 01:20:04.796157 2830 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:36046:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:20:04.803920 2830 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:20:04.831255 3289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.816536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.20.102.131:33541)
0925 01:20:04.823391 (+ 6855us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:04.823401 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:04.823433 (+ 32us) server_negotiation.cc:407] Connection header received
0925 01:20:04.823525 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:04.823530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:04.823689 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:04.823876 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:04.824453 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.826341 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.828849 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.830068 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.830282 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.830468 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:04.830744 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 01:20:04.830911 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6670,"thread_start_us":6511,"threads_started":1}
I20260925 01:20:04.831377 3288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.816258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:33541 (local address 127.0.0.1:37358)
0925 01:20:04.817158 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:04.817193 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:04.817359 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:04.823870 (+ 6511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:04.823874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:04.823899 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:04.824267 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:04.824277 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.826485 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.826491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:04.828381 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:04.828393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.830278 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.830284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:04.830467 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.830526 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 01:20:04.830688 (+ 162us) client_negotiation.cc:240] Negotiation successful
0925 01:20:04.830888 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"mutex_wait_us":3,"thread_start_us":446,"threads_started":1}
I20260925 01:20:04.832273 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.817572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36400 (local address 127.20.102.129:39795)
0925 01:20:04.818793 (+ 1221us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:04.818806 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:04.821387 (+ 2581us) server_negotiation.cc:407] Connection header received
0925 01:20:04.822535 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:04.822549 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:04.822969 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:04.823167 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:04.825040 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.826531 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.829279 (+ 2748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.830758 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.830978 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.831139 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:04.831880 (+ 741us) server_negotiation.cc:299] Negotiation successful
0925 01:20:04.832025 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":328,"threads_started":1}
I20260925 01:20:04.832269 3291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.816933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39795 (local address 127.0.0.1:36400)
0925 01:20:04.821242 (+ 4309us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:04.821266 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:04.821520 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:04.823560 (+ 2040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:04.823571 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:04.823625 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:04.824614 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:04.824628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.826656 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.826662 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:04.829088 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:04.829102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.831368 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.831378 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:04.831604 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.831696 (+ 92us) client_negotiation.cc:769] Sending connection context
0925 01:20:04.831778 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 01:20:04.831917 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4141,"mutex_wait_us":635,"thread_start_us":3965,"threads_started":1}
I20260925 01:20:04.833428 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.817493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:41643 (local address 127.0.0.1:35252)
0925 01:20:04.822139 (+ 4646us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:04.822161 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:04.822289 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:04.824802 (+ 2513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:04.824806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:04.824831 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:04.825264 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:04.825275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.827002 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.827009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:04.829277 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:04.829289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.831404 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.831410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:04.831619 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.832769 (+ 1150us) client_negotiation.cc:769] Sending connection context
0925 01:20:04.832863 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 01:20:04.833016 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4511,"mutex_wait_us":44,"thread_start_us":4350,"threads_started":1}
I20260925 01:20:04.834408 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.817908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.20.102.130:41643)
0925 01:20:04.824310 (+ 6402us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:04.824318 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:04.824349 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:20:04.824418 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:04.824423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:04.824567 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:04.824816 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:04.825479 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.826855 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.829454 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.830838 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.831005 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.831139 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:04.833941 (+ 2802us) server_negotiation.cc:299] Negotiation successful
0925 01:20:04.834127 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6254,"thread_start_us":6161,"threads_started":1}
I20260925 01:20:04.835896 2950 tablet_service.cc:1467] Processing CreateTablet for tablet c1b8ce7887504ef0aa306dba76f668ca (DEFAULT_TABLE table=test-table [id=1a96878d399445bead9e9315d7267431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:20:04.835744 3213 tablet_service.cc:1467] Processing CreateTablet for tablet c1b8ce7887504ef0aa306dba76f668ca (DEFAULT_TABLE table=test-table [id=1a96878d399445bead9e9315d7267431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:20:04.837200 2950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1b8ce7887504ef0aa306dba76f668ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:20:04.837222 3213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1b8ce7887504ef0aa306dba76f668ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:20:04.838137 3082 tablet_service.cc:1467] Processing CreateTablet for tablet c1b8ce7887504ef0aa306dba76f668ca (DEFAULT_TABLE table=test-table [id=1a96878d399445bead9e9315d7267431]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:20:04.839509 3082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1b8ce7887504ef0aa306dba76f668ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:20:04.844543 3015 heartbeater.cc:503] Master 127.20.102.190:39107 was elected leader, sending a full tablet report...
I20260925 01:20:04.851002 3294 tablet_bootstrap.cc:492] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd: Bootstrap starting.
I20260925 01:20:04.853408 3295 tablet_bootstrap.cc:492] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c: Bootstrap starting.
I20260925 01:20:04.854871 3296 tablet_bootstrap.cc:492] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679: Bootstrap starting.
I20260925 01:20:04.856034 3294 tablet_bootstrap.cc:654] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd: Neither blocks nor log segments found. Creating new log.
I20260925 01:20:04.856977 3295 tablet_bootstrap.cc:654] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c: Neither blocks nor log segments found. Creating new log.
I20260925 01:20:04.857453 3294 log.cc:824] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:04.858486 3296 tablet_bootstrap.cc:654] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679: Neither blocks nor log segments found. Creating new log.
I20260925 01:20:04.858889 3295 log.cc:824] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:04.859624 3296 log.cc:824] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:04.861467 3294 tablet_bootstrap.cc:492] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd: No bootstrap required, opened a new log
I20260925 01:20:04.861815 3294 ts_tablet_manager.cc:1397] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260925 01:20:04.862169 3295 tablet_bootstrap.cc:492] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c: No bootstrap required, opened a new log
I20260925 01:20:04.862414 3295 ts_tablet_manager.cc:1397] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 01:20:04.863257 3296 tablet_bootstrap.cc:492] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679: No bootstrap required, opened a new log
I20260925 01:20:04.863507 3296 ts_tablet_manager.cc:1397] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260925 01:20:04.869762 3295 raft_consensus.cc:348] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:04.870183 3295 raft_consensus.cc:374] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:20:04.870318 3295 raft_consensus.cc:729] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6084b2735503495a836de8d4e4bac29c, State: Initialized, Role: FOLLOWER
I20260925 01:20:04.871099 3295 consensus_queue.cc:260] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:04.872803 3294 raft_consensus.cc:348] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:04.873245 3294 raft_consensus.cc:374] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:20:04.873310 3147 heartbeater.cc:503] Master 127.20.102.190:39107 was elected leader, sending a full tablet report...
I20260925 01:20:04.873353 3294 raft_consensus.cc:729] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2743ab1f0ca42838749a0d149e0b0dd, State: Initialized, Role: FOLLOWER
I20260925 01:20:04.874221 3295 ts_tablet_manager.cc:1428] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20260925 01:20:04.874061 3294 consensus_queue.cc:260] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:04.874347 3296 raft_consensus.cc:348] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:04.874812 3296 raft_consensus.cc:374] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:20:04.874925 3296 raft_consensus.cc:729] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87696cdf3404ed880af0ebc60556679, State: Initialized, Role: FOLLOWER
I20260925 01:20:04.875523 3296 consensus_queue.cc:260] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:04.876703 3278 heartbeater.cc:503] Master 127.20.102.190:39107 was elected leader, sending a full tablet report...
I20260925 01:20:04.877377 3296 ts_tablet_manager.cc:1428] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679: Time spent starting tablet: real 0.014s user 0.005s sys 0.011s
I20260925 01:20:04.879523 3294 ts_tablet_manager.cc:1428] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
I20260925 01:20:04.934064 3300 raft_consensus.cc:482] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:20:04.934412 3300 raft_consensus.cc:504] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:04.936147 3300 leader_election.cc:288] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541), b2743ab1f0ca42838749a0d149e0b0dd (127.20.102.129:39795)
I20260925 01:20:04.946144 3304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.936471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:33541 (local address 127.20.102.130:51355)
0925 01:20:04.937621 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:04.937643 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:04.937816 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:04.938219 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:04.938224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:04.938325 (+ 101us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:04.938854 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:04.938868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.940323 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.940326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:04.943020 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:04.943037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.944953 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.944960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:04.945351 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.945410 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 01:20:04.945568 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 01:20:04.945703 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"mutex_wait_us":229,"spinlock_wait_cycles":39808,"thread_start_us":731,"threads_started":1}
I20260925 01:20:04.946376 3289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.936605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:51355 (local address 127.20.102.131:33541)
0925 01:20:04.936840 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:04.936847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:04.937758 (+ 911us) server_negotiation.cc:407] Connection header received
0925 01:20:04.937978 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:04.937983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:04.938070 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:04.938177 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:04.939036 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.940184 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.943224 (+ 3040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.944468 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.944663 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.944756 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:04.945641 (+ 885us) server_negotiation.cc:299] Negotiation successful
0925 01:20:04.945895 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:20:04.947074 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.936621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:51617 (local address 127.20.102.129:39795)
0925 01:20:04.936840 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:04.936847 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:04.937867 (+ 1020us) server_negotiation.cc:407] Connection header received
0925 01:20:04.937977 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:04.937983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:04.938083 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:04.938177 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:04.939052 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.940182 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.943224 (+ 3042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.944467 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.944642 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.944756 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:04.946641 (+ 1885us) server_negotiation.cc:299] Negotiation successful
0925 01:20:04.946813 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 01:20:04.947160 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.936625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39795 (local address 127.20.102.130:51617)
0925 01:20:04.937752 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:04.937768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:04.937870 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:04.938219 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:04.938224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:04.938263 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:04.938854 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:04.938868 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.940321 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.940326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:04.943020 (+ 2694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:04.943037 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.944957 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.944962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:04.945350 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.946431 (+ 1081us) client_negotiation.cc:769] Sending connection context
0925 01:20:04.946595 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 01:20:04.946736 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":1117,"thread_start_us":588,"threads_started":1}
I20260925 01:20:04.947780 3233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1b8ce7887504ef0aa306dba76f668ca" candidate_uuid: "6084b2735503495a836de8d4e4bac29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87696cdf3404ed880af0ebc60556679" is_pre_election: true
I20260925 01:20:04.948590 3233 raft_consensus.cc:2454] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084b2735503495a836de8d4e4bac29c in term 0.
I20260925 01:20:04.949452 3038 leader_election.cc:302] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084b2735503495a836de8d4e4bac29c, c87696cdf3404ed880af0ebc60556679; no voters:
I20260925 01:20:04.954118 3300 raft_consensus.cc:2781] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:20:04.954299 3300 raft_consensus.cc:482] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:20:04.954427 3300 raft_consensus.cc:3037] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:20:04.954849 2970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1b8ce7887504ef0aa306dba76f668ca" candidate_uuid: "6084b2735503495a836de8d4e4bac29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" is_pre_election: true
I20260925 01:20:04.955561 2970 raft_consensus.cc:2454] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6084b2735503495a836de8d4e4bac29c in term 0.
I20260925 01:20:04.957799 3300 raft_consensus.cc:504] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:04.958608 3300 leader_election.cc:288] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [CANDIDATE]: Term 1 election: Requested vote from peers c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541), b2743ab1f0ca42838749a0d149e0b0dd (127.20.102.129:39795)
I20260925 01:20:04.959112 3233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1b8ce7887504ef0aa306dba76f668ca" candidate_uuid: "6084b2735503495a836de8d4e4bac29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c87696cdf3404ed880af0ebc60556679"
I20260925 01:20:04.959259 2970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1b8ce7887504ef0aa306dba76f668ca" candidate_uuid: "6084b2735503495a836de8d4e4bac29c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2743ab1f0ca42838749a0d149e0b0dd"
I20260925 01:20:04.959401 3233 raft_consensus.cc:3037] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:20:04.959484 2970 raft_consensus.cc:3037] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:20:04.962006 2970 raft_consensus.cc:2454] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084b2735503495a836de8d4e4bac29c in term 1.
I20260925 01:20:04.962543 3036 leader_election.cc:302] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6084b2735503495a836de8d4e4bac29c, b2743ab1f0ca42838749a0d149e0b0dd; no voters:
I20260925 01:20:04.962895 3300 raft_consensus.cc:2781] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:20:04.963212 3233 raft_consensus.cc:2454] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6084b2735503495a836de8d4e4bac29c in term 1.
I20260925 01:20:04.963986 3300 raft_consensus.cc:686] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [term 1 LEADER]: Becoming Leader. State: Replica: 6084b2735503495a836de8d4e4bac29c, State: Running, Role: LEADER
I20260925 01:20:04.964483 3300 consensus_queue.cc:237] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:04.969197 2829 catalog_manager.cc:5495] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c reported cstate change: term changed from 0 to 1, leader changed from <none> to 6084b2735503495a836de8d4e4bac29c (127.20.102.130). New cstate: current_term: 1 leader_uuid: "6084b2735503495a836de8d4e4bac29c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } health_report { overall_health: UNKNOWN } } }
I20260925 01:20:04.989488 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:04.983527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.20.102.130:41643)
0925 01:20:04.983743 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:04.983749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:04.983880 (+ 131us) server_negotiation.cc:407] Connection header received
0925 01:20:04.984145 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:04.984149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:04.984276 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:04.984432 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:20:04.984849 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.985568 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.987096 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:04.987517 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:04.987664 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:04.988217 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:20:04.988864 (+ 647us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:20:04.989030 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:04.989082 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 01:20:04.989167 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:20:04.997845 3233 raft_consensus.cc:1261] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679 [term 1 FOLLOWER]: Refusing update from remote peer 6084b2735503495a836de8d4e4bac29c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:20:04.997845 2970 raft_consensus.cc:1261] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Refusing update from remote peer 6084b2735503495a836de8d4e4bac29c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:20:04.998783 3300 consensus_queue.cc:1035] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:20:04.999099 3305 consensus_queue.cc:1035] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [LEADER]: Connected to new peer: Peer: permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:20:05.002905 3279 tablet.cc:2369] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:20:05.016733 3309 mvcc.cc:205] Tried to move back new op lower bound from 7333065543661076480 to 7333065543538749440. Current Snapshot: MvccSnapshot[applied={T|T < 7333065543661076480 or (T in {7333065543661076480})}]
I20260925 01:20:05.022284 3310 mvcc.cc:205] Tried to move back new op lower bound from 7333065543661076480 to 7333065543538749440. Current Snapshot: MvccSnapshot[applied={T|T < 7333065543661076480}]
I20260925 01:20:05.029695 2830 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:36046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:20:05.029976 2830 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:36046:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:20:05.033213 2830 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Sending DeleteTablet for 3 replicas of tablet c1b8ce7887504ef0aa306dba76f668ca
I20260925 01:20:05.034479 3082 tablet_service.cc:1514] Processing DeleteTablet for tablet c1b8ce7887504ef0aa306dba76f668ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:20:05 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:35252
I20260925 01:20:05.034603 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 2867
I20260925 01:20:05.034484 2950 tablet_service.cc:1514] Processing DeleteTablet for tablet c1b8ce7887504ef0aa306dba76f668ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:20:05 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:36400
I20260925 01:20:05.035610 3213 tablet_service.cc:1514] Processing DeleteTablet for tablet c1b8ce7887504ef0aa306dba76f668ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:20:05 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:37358
I20260925 01:20:05.050836 3320 tablet_replica.cc:330] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679: stopping tablet replica
I20260925 01:20:05.051399 3320 raft_consensus.cc:2229] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:20:05.052079 3320 raft_consensus.cc:2258] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:20:05.054489 3290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:05.035793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36402 (local address 127.20.102.129:39795)
0925 01:20:05.035972 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:05.035980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:05.036016 (+ 36us) server_negotiation.cc:407] Connection header received
0925 01:20:05.036156 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:05.036164 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:05.036278 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:05.036401 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:05.038229 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:05.039707 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:05.041656 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:05.042550 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:05.042703 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:05.046113 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:05.046132 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:05.046159 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:05.046208 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:05.051053 (+ 4845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:05.052023 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:05.052032 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:05.052037 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:05.052212 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:05.052597 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:05.052601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:05.052605 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:05.053153 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:05.053460 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:05.053861 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 01:20:05.054128 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:20:05.054574 3320 ts_tablet_manager.cc:1905] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:20:05.055253 3010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:05.057286 3318 tablet_replica.cc:330] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c: stopping tablet replica
I20260925 01:20:05.057744 3318 raft_consensus.cc:2229] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:20:05.059659 3318 raft_consensus.cc:2258] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:20:05.063064 3318 ts_tablet_manager.cc:1905] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:20:05.064051 3320 ts_tablet_manager.cc:1918] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:20:05.064213 3320 log.cc:1196] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/c1b8ce7887504ef0aa306dba76f668ca
I20260925 01:20:05.064843 3320 ts_tablet_manager.cc:1939] T c1b8ce7887504ef0aa306dba76f668ca P c87696cdf3404ed880af0ebc60556679: Deleting consensus metadata
I20260925 01:20:05.067240 2817 catalog_manager.cc:4841] TS c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541): tablet c1b8ce7887504ef0aa306dba76f668ca (table test-table [id=1a96878d399445bead9e9315d7267431]) successfully deleted
I20260925 01:20:05.072984 3318 ts_tablet_manager.cc:1918] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:20:05.073268 3318 log.cc:1196] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/c1b8ce7887504ef0aa306dba76f668ca
I20260925 01:20:05.074115 3318 ts_tablet_manager.cc:1939] T c1b8ce7887504ef0aa306dba76f668ca P 6084b2735503495a836de8d4e4bac29c: Deleting consensus metadata
I20260925 01:20:05.076175 2817 catalog_manager.cc:4841] TS 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643): tablet c1b8ce7887504ef0aa306dba76f668ca (table test-table [id=1a96878d399445bead9e9315d7267431]) successfully deleted
I20260925 01:20:05.166702 3319 tablet_replica.cc:330] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd: stopping tablet replica
I20260925 01:20:05.167304 3319 raft_consensus.cc:2229] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:20:05.167726 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 2867
W20260925 01:20:05.186357 2815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.102.129:39795 (error 108)
I20260925 01:20:05.186506 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 3019
I20260925 01:20:05.188648 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:05.188183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39795 (local address 127.0.0.1:36412)
0925 01:20:05.188380 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:05.188510 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 01:20:05.188890 2815 catalog_manager.cc:4584] TS b2743ab1f0ca42838749a0d149e0b0dd (127.20.102.129:39795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c1b8ce7887504ef0aa306dba76f668ca: Network error: Client connection negotiation failed: client connection to 127.20.102.129:39795: connect: Connection refused (error 111)
I20260925 01:20:05.210480 3292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:05.187581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.20.102.130:41643)
0925 01:20:05.187878 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:05.187885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:05.187909 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:20:05.188104 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:05.188109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:05.188172 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:05.188280 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:05.190859 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:05.191706 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:05.193859 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:05.194353 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:05.194469 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:05.203601 (+ 9132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:05.203618 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:05.203635 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:05.203684 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:05.207231 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:05.208323 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:05.208332 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:05.208334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:05.208484 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:05.208948 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:05.208951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:05.208954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:05.209322 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:05.209592 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:05.209967 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 01:20:05.210156 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139}
I20260925 01:20:05.211210 3142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:05.242691 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:05.242093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39795 (local address 127.0.0.1:36414)
0925 01:20:05.242344 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:05.242482 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 01:20:05.243837 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:05.243471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39795 (local address 127.0.0.1:36426)
0925 01:20:05.243655 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:05.243737 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 01:20:05.304661 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:05.304121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39795 (local address 127.0.0.1:36432)
0925 01:20:05.304359 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:05.304464 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 01:20:05.305860 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:05.305442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39795 (local address 127.0.0.1:36434)
0925 01:20:05.305615 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:05.305704 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 01:20:05.330623 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 3019
I20260925 01:20:05.352106 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 3150
I20260925 01:20:05.366914 3289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:05.353014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37368 (local address 127.20.102.131:33541)
0925 01:20:05.353219 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:05.353224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:05.353293 (+ 69us) server_negotiation.cc:407] Connection header received
0925 01:20:05.353459 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:05.353463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:05.353582 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:05.353667 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:05.355193 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:05.355968 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:05.357349 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:05.357779 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:05.357873 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:05.361026 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:05.361042 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:05.361059 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:05.361099 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:05.363957 (+ 2858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:05.364825 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:05.364833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:05.364835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:05.364982 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:05.365384 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:05.365388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:05.365390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:05.365773 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:05.365999 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:05.366362 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 01:20:05.366543 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:20:05.367811 3273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:05.387019 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:05.386216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39795 (local address 127.0.0.1:36438)
0925 01:20:05.386599 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:05.386765 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 01:20:05.388173 3293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:05.387766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39795 (local address 127.0.0.1:36448)
0925 01:20:05.387985 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:05.388060 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:39795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260925 01:20:05.459925 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 3150
I20260925 01:20:05.478258 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 2797
I20260925 01:20:05.479490 2860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:05.597792 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 2797
I20260925 01:20:05.620608 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39107
--webserver_interface=127.20.102.190
--webserver_port=38767
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:39107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:20:05.976073 3326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:05.976373 3326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:05.976558 3326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:05.984097 3326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:20:05.984212 3326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:05.984313 3326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:05.984401 3326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:20:05.984495 3326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:20:05.994890 3326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:39107
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=38767
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:05.996300 3326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:05.997768 3326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:06.009954 3331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:06.010039 3332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:06.010020 3334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:06.011502 3326 server_base.cc:1034] running on GCE node
I20260925 01:20:06.012452 3326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:06.014045 3326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:06.015275 3326 hybrid_clock.cc:648] HybridClock initialized: now 1790299206015221 us; error 37 us; skew 500 ppm
I20260925 01:20:06.015661 3326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:06 dist-test-slave-s12s krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299206, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:06.018931 3326 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:20:06.020596 3326 webserver.cc:460] Webserver started at http://127.20.102.190:38767/ using document root <none> and password file <none>
I20260925 01:20:06.021395 3326 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:06.021551 3326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:06.027467 3326 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 01:20:06.030436 3341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:06.031790 3326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:20:06.031984 3326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "d75889f2ac7844a19913ab0b9c10864c"
format_stamp: "Formatted at 2026-09-25 01:19:59 on dist-test-slave-s12s"
server_key: "af7b1a735fe152b2fd731050e346cc26"
server_key_iv: "5d3c961c36bb72ec3721df003adb427b"
server_key_version: "encryptionkey@0"
I20260925 01:20:06.033054 3326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:06.057811 3326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:06.061796 3326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:06.062093 3326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:06.081859 3326 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:39107
I20260925 01:20:06.081879 3392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:39107 every 8 connection(s)
I20260925 01:20:06.083467 3326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:20:06.086076 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 3326
I20260925 01:20:06.086910 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:39795
--local_ip_for_outbound_sockets=127.20.102.129
--tserver_master_addrs=127.20.102.190:39107
--webserver_port=36791
--webserver_interface=127.20.102.129
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:20:06.090785 3393 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:20:06.094233 3393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Bootstrap starting.
I20260925 01:20:06.115505 3393 log.cc:824] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:06.132140 3393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:20:06.132827 3393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c: Bootstrap complete.
I20260925 01:20:06.143272 3393 raft_consensus.cc:348] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } }
I20260925 01:20:06.144172 3393 raft_consensus.cc:729] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75889f2ac7844a19913ab0b9c10864c, State: Initialized, Role: FOLLOWER
I20260925 01:20:06.144950 3393 consensus_queue.cc:260] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } }
I20260925 01:20:06.145190 3393 raft_consensus.cc:388] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:20:06.145308 3393 raft_consensus.cc:482] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:20:06.145480 3393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 2 FOLLOWER]: Advancing to term 3
I20260925 01:20:06.149446 3393 raft_consensus.cc:504] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } }
I20260925 01:20:06.149959 3393 leader_election.cc:302] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d75889f2ac7844a19913ab0b9c10864c; no voters:
I20260925 01:20:06.150651 3393 leader_election.cc:288] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 01:20:06.150794 3397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 3 FOLLOWER]: Leader election won for term 3
I20260925 01:20:06.152069 3397 raft_consensus.cc:686] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [term 3 LEADER]: Becoming Leader. State: Replica: d75889f2ac7844a19913ab0b9c10864c, State: Running, Role: LEADER
I20260925 01:20:06.152634 3397 consensus_queue.cc:237] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } }
I20260925 01:20:06.153053 3393 sys_catalog.cc:564] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:20:06.155395 3399 sys_catalog.cc:455] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d75889f2ac7844a19913ab0b9c10864c. Latest consensus state: current_term: 3 leader_uuid: "d75889f2ac7844a19913ab0b9c10864c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } } }
I20260925 01:20:06.155480 3398 sys_catalog.cc:455] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d75889f2ac7844a19913ab0b9c10864c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75889f2ac7844a19913ab0b9c10864c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39107 } } }
I20260925 01:20:06.155797 3399 sys_catalog.cc:458] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: This master's current role is: LEADER
I20260925 01:20:06.155900 3398 sys_catalog.cc:458] T 00000000000000000000000000000000 P d75889f2ac7844a19913ab0b9c10864c [sys.catalog]: This master's current role is: LEADER
I20260925 01:20:06.156801 3402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:20:06.177856 3402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:20:06.179415 3402 catalog_manager.cc:1259] Loaded cluster ID: 245edc67f55049afa764572a83f936e6
I20260925 01:20:06.179618 3402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 01:20:06.181639 3402 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f83499719ad google::LogMessage::Fail() at ??:0
@ 0x7f834997488a google::LogMessage::SendToLog() at ??:0
@ 0x7f83499714de google::LogMessage::Flush() at ??:0
@ 0x7f8349972a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f835f3a4512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f835f3a1976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f835f429610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f835f429439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f834c5bd345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f834c5c091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f834c5c0799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f834c59beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f834f8756db start_thread at ??:0
@ 0x7f834880b61f clone at ??:0
W20260925 01:20:06.529340 3395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:06.529698 3395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:06.529870 3395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:06.537468 3395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:06.537683 3395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:06.537873 3395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:20:06.548192 3395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:39795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=36791
--tserver_master_addrs=127.20.102.190:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:06.549712 3395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:06.551186 3395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:06.564967 3434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:06.565194 3436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:06.565341 3433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:06.566524 3395 server_base.cc:1034] running on GCE node
I20260925 01:20:06.567327 3395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:06.568586 3395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:06.569867 3395 hybrid_clock.cc:648] HybridClock initialized: now 1790299206569811 us; error 48 us; skew 500 ppm
I20260925 01:20:06.570246 3395 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:06 dist-test-slave-s12s krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299206, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:06.573937 3395 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:20:06.576149 3395 webserver.cc:460] Webserver started at http://127.20.102.129:36791/ using document root <none> and password file <none>
I20260925 01:20:06.576953 3395 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:06.577090 3395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:06.582880 3395 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 01:20:06.586030 3443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:06.587463 3395 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:20:06.587651 3395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "b2743ab1f0ca42838749a0d149e0b0dd"
format_stamp: "Formatted at 2026-09-25 01:20:00 on dist-test-slave-s12s"
server_key: "86ad9d0cffe3187d25d17568232c381a"
server_key_iv: "96d4be974af53a8fb09a59294273edc5"
server_key_version: "encryptionkey@0"
I20260925 01:20:06.588665 3395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:06.607888 3395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:06.612515 3395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:06.612902 3395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:06.614056 3395 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:06.615948 3450 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260925 01:20:06.621963 3395 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 01:20:06.622071 3395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260925 01:20:06.622228 3395 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 01:20:06.625741 3395 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 01:20:06.625834 3395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.003s sys 0.000s
I20260925 01:20:06.626233 3450 tablet_bootstrap.cc:492] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd: Bootstrap starting.
I20260925 01:20:06.661566 3450 log.cc:824] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:06.667049 3395 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:39795
I20260925 01:20:06.667086 3557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:39795 every 8 connection(s)
I20260925 01:20:06.668650 3395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:20:06.673450 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 3395
I20260925 01:20:06.674288 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:41643
--local_ip_for_outbound_sockets=127.20.102.130
--tserver_master_addrs=127.20.102.190:39107
--webserver_port=42735
--webserver_interface=127.20.102.130
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:20:06.674629 3560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:06.671822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:41039)
0925 01:20:06.672802 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:06.673195 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":405,"threads_started":1}
W20260925 01:20:06.675365 3558 heartbeater.cc:644] Failed to heartbeat to 127.20.102.190:39107 (0 consecutive failures): Network error: Failed to ping master at 127.20.102.190:39107: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
I20260925 01:20:06.676313 3450 tablet_bootstrap.cc:492] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:20:06.676631 3560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:06.676020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:58901)
0925 01:20:06.676209 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:06.676303 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 01:20:06.677058 3450 tablet_bootstrap.cc:492] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd: Bootstrap complete.
I20260925 01:20:06.677851 3560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:06.677372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:35193)
0925 01:20:06.677559 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:06.677673 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 01:20:06.678205 3558 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 01:20:06.678682 3450 ts_tablet_manager.cc:1397] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd: Time spent bootstrapping tablet: real 0.053s user 0.035s sys 0.015s
I20260925 01:20:06.685468 3450 raft_consensus.cc:348] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:06.686415 3450 raft_consensus.cc:729] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2743ab1f0ca42838749a0d149e0b0dd, State: Initialized, Role: FOLLOWER
I20260925 01:20:06.687100 3450 consensus_queue.cc:260] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:06.688791 3450 ts_tablet_manager.cc:1428] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd: Time spent starting tablet: real 0.010s user 0.010s sys 0.002s
I20260925 01:20:06.689137 3560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:06.688650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:42821)
0925 01:20:06.688881 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:06.688980 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 01:20:06.690100 3560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:06.689719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:58061)
0925 01:20:06.689870 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:06.689947 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 01:20:07.028560 3561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:07.028910 3561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:07.029014 3561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:07.036269 3561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:07.036382 3561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:07.036506 3561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:20:07.046380 3561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:41643
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=42735
--tserver_master_addrs=127.20.102.190:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:07.047668 3561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:07.049027 3561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:07.062678 3567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:07.062729 3570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:07.062615 3568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:07.066222 3561 server_base.cc:1034] running on GCE node
I20260925 01:20:07.066869 3561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:07.067875 3561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:07.069131 3561 hybrid_clock.cc:648] HybridClock initialized: now 1790299207069084 us; error 29 us; skew 500 ppm
I20260925 01:20:07.069482 3561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:07 dist-test-slave-s12s krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299207, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:07.072765 3561 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:20:07.074278 3561 webserver.cc:460] Webserver started at http://127.20.102.130:42735/ using document root <none> and password file <none>
I20260925 01:20:07.075017 3561 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:07.075174 3561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:07.080861 3561 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:20:07.083674 3577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:07.085057 3561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:20:07.085263 3561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "6084b2735503495a836de8d4e4bac29c"
format_stamp: "Formatted at 2026-09-25 01:20:00 on dist-test-slave-s12s"
server_key: "739a70249514a27589ac720de6fdd523"
server_key_iv: "3f0cc031a5d8468dcbb7ed1571573540"
server_key_version: "encryptionkey@0"
I20260925 01:20:07.086269 3561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:07.104547 3561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:07.109229 3561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:07.109468 3561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:07.110526 3561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:07.111924 3561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:20:07.112012 3561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:07.112129 3561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:20:07.112208 3561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:07.146991 3561 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:41643
I20260925 01:20:07.147099 3689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:41643 every 8 connection(s)
I20260925 01:20:07.148574 3561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:20:07.150957 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 3561
I20260925 01:20:07.151589 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:33541
--local_ip_for_outbound_sockets=127.20.102.131
--tserver_master_addrs=127.20.102.190:39107
--webserver_port=41625
--webserver_interface=127.20.102.131
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:20:07.153950 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:07.151923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:46041)
0925 01:20:07.152785 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:07.153056 (+ 271us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":356,"threads_started":1}
W20260925 01:20:07.154978 3690 heartbeater.cc:644] Failed to heartbeat to 127.20.102.190:39107 (0 consecutive failures): Network error: Failed to ping master at 127.20.102.190:39107: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
I20260925 01:20:07.155989 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:07.155633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:51493)
0925 01:20:07.155797 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:07.155873 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 01:20:07.157040 3692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:07.156619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:42469)
0925 01:20:07.156832 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:07.156914 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
W20260925 01:20:07.157316 3690 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:20:07.490546 3693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:07.490909 3693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:07.491032 3693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:07.498690 3693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:07.498799 3693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:07.498910 3693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:20:07.509163 3693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:33541
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=41625
--tserver_master_addrs=127.20.102.190:39107
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:07.510540 3693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:07.511880 3693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:07.525713 3698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:07.525846 3699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:07.525807 3701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:07.526703 3693 server_base.cc:1034] running on GCE node
I20260925 01:20:07.527315 3693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:07.528296 3693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:07.529568 3693 hybrid_clock.cc:648] HybridClock initialized: now 1790299207529479 us; error 72 us; skew 500 ppm
I20260925 01:20:07.529970 3693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:07 dist-test-slave-s12s krb5kdc[2245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299207, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:07.533397 3693 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:20:07.535040 3693 webserver.cc:460] Webserver started at http://127.20.102.131:41625/ using document root <none> and password file <none>
I20260925 01:20:07.535842 3693 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:07.536005 3693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:07.541913 3693 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 01:20:07.545312 3708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:07.546765 3693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:20:07.546952 3693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "c87696cdf3404ed880af0ebc60556679"
format_stamp: "Formatted at 2026-09-25 01:20:01 on dist-test-slave-s12s"
server_key: "97b7852bcaa68b4a7dc75d3b9a118ff3"
server_key_iv: "c8068911948dcb65cfbd18431ee3865b"
server_key_version: "encryptionkey@0"
I20260925 01:20:07.547973 3693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:07.575512 3693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:07.579936 3693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:07.580276 3693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:07.581434 3693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:07.582818 3693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:20:07.582964 3693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:07.583104 3693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:20:07.583199 3693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:07.618310 3693 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:33541
I20260925 01:20:07.618398 3820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:33541 every 8 connection(s)
I20260925 01:20:07.619819 3693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:20:07.625305 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:07.622985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:43241)
0925 01:20:07.623923 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:07.624259 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":444,"threads_started":1}
W20260925 01:20:07.626083 3821 heartbeater.cc:644] Failed to heartbeat to 127.20.102.190:39107 (0 consecutive failures): Network error: Failed to ping master at 127.20.102.190:39107: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
I20260925 01:20:07.627022 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:07.626655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:36637)
0925 01:20:07.626832 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:07.626909 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 01:20:07.627408 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 3693
I20260925 01:20:07.628039 3823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:07.627694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:56037)
0925 01:20:07.627842 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:07.627928 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 01:20:07.628363 3821 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 01:20:07.691825 3825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:07.690994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:42403)
0925 01:20:07.691513 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:07.691602 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":253,"threads_started":1}
I20260925 01:20:07.987035 3826 raft_consensus.cc:482] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:20:07.987411 3826 raft_consensus.cc:504] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:07.989243 3826 leader_election.cc:288] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541), 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643)
Sep 25 01:20:07 dist-test-slave-s12s krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299206, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
Sep 25 01:20:08 dist-test-slave-s12s krb5kdc[2245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299206, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:20:08.014312 3827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:07.989946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:41643 (local address 127.20.102.129:59743)
0925 01:20:07.991831 (+ 1885us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:07.991860 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:07.992011 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:07.994698 (+ 2687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:07.994704 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:07.995555 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:20:07.996310 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:07.996335 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:07.998050 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:07.998058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:07.999545 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:07.999559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:08.001397 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:08.001406 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:08.001659 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:08.002156 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:20:08.002190 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:20:08.005506 (+ 3316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:20:08.009509 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:08.009521 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:08.009540 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:08.010232 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:08.010765 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:08.010770 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:08.010774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:08.011121 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:08.012283 (+ 1162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:20:08.012292 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:20:08.012841 (+ 549us) client_negotiation.cc:769] Sending connection context
0925 01:20:08.013184 (+ 343us) client_negotiation.cc:240] Negotiation successful
0925 01:20:08.013769 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1759,"mutex_wait_us":235,"spinlock_wait_cycles":34176,"thread_start_us":1569,"threads_started":1}
I20260925 01:20:08.014329 3829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:07.990317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:59743 (local address 127.20.102.130:41643)
0925 01:20:07.992117 (+ 1800us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:07.992128 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:07.992183 (+ 55us) server_negotiation.cc:407] Connection header received
0925 01:20:07.993968 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:07.993977 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:07.994327 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:07.994726 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:07.996556 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:07.997820 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:07.999720 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:08.000863 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:08.001057 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:08.005900 (+ 4843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:08.005915 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:08.005944 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:08.005988 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:08.009310 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:08.010388 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:08.010398 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:08.010407 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:08.010629 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:08.011230 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:08.011238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:08.011242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:08.012105 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:08.012314 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:08.013461 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0925 01:20:08.013907 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1633,"thread_start_us":1517,"threads_started":1}
I20260925 01:20:08.014309 3825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:07.989505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:33541 (local address 127.20.102.129:48121)
0925 01:20:07.989768 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:07.989819 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:07.990517 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:07.994646 (+ 4129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:07.994657 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:07.995554 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:20:07.996310 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:07.996331 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:07.998075 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:07.998086 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:07.999522 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:07.999565 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:08.000944 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:08.000958 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:08.001259 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:08.002143 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:20:08.002182 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:20:08.005748 (+ 3566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:20:08.010081 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:08.010086 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:08.010089 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:08.010591 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:08.011161 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:08.011166 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:08.011169 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:08.011343 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:08.012283 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:20:08.012292 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:20:08.012841 (+ 549us) client_negotiation.cc:769] Sending connection context
0925 01:20:08.013184 (+ 343us) client_negotiation.cc:240] Negotiation successful
0925 01:20:08.013890 (+ 706us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":408,"spinlock_wait_cycles":18688}
I20260925 01:20:08.014348 3828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:07.989983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:48121 (local address 127.20.102.131:33541)
0925 01:20:07.991627 (+ 1644us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:07.991645 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:07.991700 (+ 55us) server_negotiation.cc:407] Connection header received
0925 01:20:07.993715 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:07.993725 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:07.994158 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:07.994450 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:07.996539 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:07.997874 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:07.999744 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:08.000534 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:08.000758 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:08.006092 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:08.006100 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:08.006114 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:08.006163 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:08.009919 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:08.010765 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:08.010775 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:08.010784 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:08.010993 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:08.011481 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:08.011487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:08.011490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:08.012105 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:08.012314 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:08.013461 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0925 01:20:08.013888 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1375,"thread_start_us":1159,"threads_started":1}
W20260925 01:20:08.017926 3447 leader_election.cc:341] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541): Not found: Tablet not found: c1b8ce7887504ef0aa306dba76f668ca
W20260925 01:20:08.018255 3447 leader_election.cc:341] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643): Not found: Tablet not found: c1b8ce7887504ef0aa306dba76f668ca
I20260925 01:20:08.018395 3447 leader_election.cc:302] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2743ab1f0ca42838749a0d149e0b0dd; no voters: 6084b2735503495a836de8d4e4bac29c, c87696cdf3404ed880af0ebc60556679
I20260925 01:20:08.018841 3826 raft_consensus.cc:2726] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 01:20:08.158921 3830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:08.157942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:45871)
0925 01:20:08.158465 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:08.158664 (+ 199us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":238,"threads_started":1}
I20260925 01:20:08.629809 3831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:08.628950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:43437)
0925 01:20:08.629434 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:08.629587 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":226,"threads_started":1}
I20260925 01:20:08.693728 3832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:08.692953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:45037)
0925 01:20:08.693473 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:08.693566 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":299,"threads_started":1}
I20260925 01:20:09.160833 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:09.159948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:54511)
0925 01:20:09.160447 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:09.160575 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":278,"threads_started":1}
I20260925 01:20:09.631451 3834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:09.630749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:47035)
0925 01:20:09.631184 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:09.631270 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":227,"threads_started":1}
I20260925 01:20:09.695595 3835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:09.694715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:60713)
0925 01:20:09.695298 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:09.695399 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":256,"threads_started":1}
I20260925 01:20:10.162751 3836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:10.161964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:33739)
0925 01:20:10.162478 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:10.162578 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":265,"threads_started":1}
I20260925 01:20:10.226138 3837 raft_consensus.cc:482] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:20:10.226372 3837 raft_consensus.cc:504] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:10.227093 3837 leader_election.cc:288] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541), 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643)
W20260925 01:20:10.228173 3447 leader_election.cc:341] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541): Not found: Tablet not found: c1b8ce7887504ef0aa306dba76f668ca
W20260925 01:20:10.228407 3447 leader_election.cc:341] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643): Not found: Tablet not found: c1b8ce7887504ef0aa306dba76f668ca
I20260925 01:20:10.228518 3447 leader_election.cc:302] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2743ab1f0ca42838749a0d149e0b0dd; no voters: 6084b2735503495a836de8d4e4bac29c, c87696cdf3404ed880af0ebc60556679
I20260925 01:20:10.228859 3837 raft_consensus.cc:2726] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 01:20:10.633292 3838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:10.632445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:43425)
0925 01:20:10.633011 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:10.633113 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":335,"threads_started":1}
I20260925 01:20:10.697433 3839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:10.696569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:39605)
0925 01:20:10.697126 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:10.697232 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":284,"threads_started":1}
I20260925 01:20:11.164572 3840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:11.163786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:58679)
0925 01:20:11.164275 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:11.164371 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":233,"threads_started":1}
I20260925 01:20:11.635138 3841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:11.634277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:33189)
0925 01:20:11.634846 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:11.634940 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":303,"threads_started":1}
I20260925 01:20:11.699301 3842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:11.698577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:59483)
0925 01:20:11.699017 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:11.699118 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":229,"threads_started":1}
I20260925 01:20:12.166720 3843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:12.165734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:50139)
0925 01:20:12.166323 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:12.166425 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":297,"threads_started":1}
I20260925 01:20:12.321085 3844 raft_consensus.cc:482] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:20:12.321338 3844 raft_consensus.cc:504] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:12.322099 3844 leader_election.cc:288] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541), 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643)
W20260925 01:20:12.323264 3447 leader_election.cc:341] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541): Not found: Tablet not found: c1b8ce7887504ef0aa306dba76f668ca
W20260925 01:20:12.323503 3447 leader_election.cc:341] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643): Not found: Tablet not found: c1b8ce7887504ef0aa306dba76f668ca
I20260925 01:20:12.323681 3447 leader_election.cc:302] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2743ab1f0ca42838749a0d149e0b0dd; no voters: 6084b2735503495a836de8d4e4bac29c, c87696cdf3404ed880af0ebc60556679
I20260925 01:20:12.323966 3844 raft_consensus.cc:2726] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 01:20:12.637125 3845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:12.636250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:40899)
0925 01:20:12.636850 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:12.636956 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":292,"threads_started":1}
I20260925 01:20:12.701208 3846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:12.700284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:40195)
0925 01:20:12.700865 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:12.700967 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":260,"threads_started":1}
I20260925 01:20:13.168846 3847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:13.167833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:54411)
0925 01:20:13.168459 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:13.168575 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":334,"threads_started":1}
I20260925 01:20:13.638972 3848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:13.638133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:39299)
0925 01:20:13.638689 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:13.638779 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":284,"threads_started":1}
I20260925 01:20:13.703033 3849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:13.702229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:33649)
0925 01:20:13.702727 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:13.702833 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":266,"threads_started":1}
I20260925 01:20:14.170825 3850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:14.169949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:50557)
0925 01:20:14.170545 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:14.170635 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":273,"threads_started":1}
I20260925 01:20:14.641340 3851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:14.640080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:49629)
0925 01:20:14.640905 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:14.641047 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":307,"threads_started":1}
I20260925 01:20:14.704932 3852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:14.704087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:47423)
0925 01:20:14.704584 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:14.704749 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":237,"threads_started":1}
I20260925 01:20:15.172658 3853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:15.171885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:54493)
0925 01:20:15.172366 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:15.172453 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":252,"threads_started":1}
I20260925 01:20:15.600446 3854 raft_consensus.cc:482] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:20:15.600672 3854 raft_consensus.cc:504] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:15.601365 3854 leader_election.cc:288] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541), 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643)
W20260925 01:20:15.602480 3447 leader_election.cc:341] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541): Not found: Tablet not found: c1b8ce7887504ef0aa306dba76f668ca
W20260925 01:20:15.602687 3447 leader_election.cc:341] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643): Not found: Tablet not found: c1b8ce7887504ef0aa306dba76f668ca
I20260925 01:20:15.602763 3447 leader_election.cc:302] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2743ab1f0ca42838749a0d149e0b0dd; no voters: 6084b2735503495a836de8d4e4bac29c, c87696cdf3404ed880af0ebc60556679
I20260925 01:20:15.602990 3854 raft_consensus.cc:2726] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 01:20:15.643100 3855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:15.642365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:58381)
0925 01:20:15.642798 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:15.642886 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":235,"threads_started":1}
I20260925 01:20:15.706948 3856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:15.705999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:56905)
0925 01:20:15.706592 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:15.706726 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":321,"threads_started":1}
I20260925 01:20:16.174614 3857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:16.173755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:33889)
0925 01:20:16.174319 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:16.174417 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":280,"threads_started":1}
I20260925 01:20:16.645012 3858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:16.644076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:45745)
0925 01:20:16.644742 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:16.644852 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":306,"threads_started":1}
I20260925 01:20:16.708803 3859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:16.707994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:33527)
0925 01:20:16.708458 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:16.708548 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":245,"threads_started":1}
I20260925 01:20:17.176663 3860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:17.175858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:54381)
0925 01:20:17.176350 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:17.176442 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":266,"threads_started":1}
I20260925 01:20:17.646621 3861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:17.645928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:39069)
0925 01:20:17.646357 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:17.646436 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":229,"threads_started":1}
I20260925 01:20:17.710754 3862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:17.709808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:54787)
0925 01:20:17.710351 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:17.710483 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":315,"threads_started":1}
I20260925 01:20:18.178725 3863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:18.177840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:55465)
0925 01:20:18.178391 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:18.178488 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":265,"threads_started":1}
I20260925 01:20:18.522122 3864 raft_consensus.cc:482] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:20:18.522398 3864 raft_consensus.cc:504] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c87696cdf3404ed880af0ebc60556679" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33541 } } peers { permanent_uuid: "6084b2735503495a836de8d4e4bac29c" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 41643 } } peers { permanent_uuid: "b2743ab1f0ca42838749a0d149e0b0dd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39795 } }
I20260925 01:20:18.523221 3864 leader_election.cc:288] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541), 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643)
W20260925 01:20:18.524269 3447 leader_election.cc:341] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c87696cdf3404ed880af0ebc60556679 (127.20.102.131:33541): Not found: Tablet not found: c1b8ce7887504ef0aa306dba76f668ca
W20260925 01:20:18.524466 3447 leader_election.cc:341] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6084b2735503495a836de8d4e4bac29c (127.20.102.130:41643): Not found: Tablet not found: c1b8ce7887504ef0aa306dba76f668ca
I20260925 01:20:18.524561 3447 leader_election.cc:302] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b2743ab1f0ca42838749a0d149e0b0dd; no voters: 6084b2735503495a836de8d4e4bac29c, c87696cdf3404ed880af0ebc60556679
I20260925 01:20:18.524855 3864 raft_consensus.cc:2726] T c1b8ce7887504ef0aa306dba76f668ca P b2743ab1f0ca42838749a0d149e0b0dd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 01:20:18.648324 3865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:18.647559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:34495)
0925 01:20:18.648076 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:18.648161 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":246,"threads_started":1}
I20260925 01:20:18.712507 3866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:18.711744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:46495)
0925 01:20:18.712225 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:18.712309 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":233,"threads_started":1}
I20260925 01:20:19.180593 3867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:19.179830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:53293)
0925 01:20:19.180311 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:19.180397 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":254,"threads_started":1}
I20260925 01:20:19.650090 3868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:19.649311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:51379)
0925 01:20:19.649814 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:19.649901 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":279,"threads_started":1}
I20260925 01:20:19.714304 3869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:19.713511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:43983)
0925 01:20:19.714007 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:19.714096 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":235,"threads_started":1}
I20260925 01:20:20.182523 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:20.181695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:44445)
0925 01:20:20.182191 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:20.182307 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":251,"threads_started":1}
I20260925 01:20:20.651911 3871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:20.651123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:58679)
0925 01:20:20.651635 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:20.651741 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":250,"threads_started":1}
I20260925 01:20:20.716329 3872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:20.715507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:33501)
0925 01:20:20.716045 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:20.716145 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":273,"threads_started":1}
I20260925 01:20:21.184531 3873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:21.183620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:52777)
0925 01:20:21.184209 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:21.184311 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":286,"threads_started":1}
I20260925 01:20:21.653810 3874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:21.653017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:51423)
0925 01:20:21.653495 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:21.653611 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":230,"threads_started":1}
I20260925 01:20:21.718544 3875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:21.717783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.129:43929)
0925 01:20:21.718263 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:21.718369 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":231,"threads_started":1}
I20260925 01:20:22.186770 3876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:22.185916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.130:35175)
0925 01:20:22.186438 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:22.186591 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":258,"threads_started":1}
I20260925 01:20:22.630380 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 3395
I20260925 01:20:22.654340 3877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:22.631506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.20.102.129:39795)
0925 01:20:22.632176 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:22.632187 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:22.632221 (+ 34us) server_negotiation.cc:407] Connection header received
0925 01:20:22.632301 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:22.632310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:22.632619 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:22.632836 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:22.635678 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:22.636801 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:22.638215 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:22.638814 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:22.638924 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:22.647586 (+ 8662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:22.647603 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:22.647620 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:22.647660 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:22.650728 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:22.651640 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:22.651648 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:22.651650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:22.651823 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:22.652309 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:22.652316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:22.652320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:22.652889 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:22.653263 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:22.653630 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 01:20:22.653953 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":292,"threads_started":1}
I20260925 01:20:22.655803 3878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:22.655024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:39107 (local address 127.20.102.131:42365)
0925 01:20:22.655527 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:22.655650 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:39107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":259,"threads_started":1}
I20260925 01:20:22.656039 3553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:22.748818 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 3395
I20260925 01:20:22.767927 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 3561
I20260925 01:20:22.782130 3881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:22.768838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57408 (local address 127.20.102.130:41643)
0925 01:20:22.769360 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:22.769365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:22.769381 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:20:22.769439 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:22.769443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:22.769503 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:22.769634 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:22.771104 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:22.771916 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:22.773225 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:22.773684 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:22.773824 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:22.776425 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:22.776430 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:22.776433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:22.776464 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:22.779239 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:22.779999 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:22.780007 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:22.780011 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:22.780152 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:22.780543 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:22.780550 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:22.780555 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:22.780886 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:22.781175 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:22.781485 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 01:20:22.781729 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":295,"threads_started":1}
I20260925 01:20:22.782912 3685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:22.868583 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 3561
I20260925 01:20:22.887127 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 3693
I20260925 01:20:22.901835 3883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:22.888081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.20.102.131:33541)
0925 01:20:22.888671 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:22.888722 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:22.888744 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:20:22.888800 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:22.888803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:22.888873 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:22.888994 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:22.890469 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:22.891297 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:22.892562 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:22.893063 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:22.893210 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:22.895974 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:22.895980 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:22.895983 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:22.896017 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:22.899079 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:22.899801 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:22.899808 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:22.899812 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:22.899951 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:22.900325 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:22.900330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:22.900334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:22.900600 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:22.901065 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:22.901213 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 01:20:22.901438 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":324,"threads_started":1}
I20260925 01:20:22.902614 3816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:22.991971 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 3693
2026-09-25T01:20:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24919 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:20:23 dist-test-slave-s12s krb5kdc[3887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:20:23 dist-test-slave-s12s krb5kdc[3887](info): set up 2 sockets
Sep 25 01:20:23 dist-test-slave-s12s krb5kdc[3887](info): commencing operation
krb5kdc: starting...
W20260925 01:20:24.101446 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:20:24 dist-test-slave-s12s krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:20:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:20:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:20:24.351935 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:34573
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:34573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:20:24.705813 3900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:24.706199 3900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:24.706375 3900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:24.713900 3900 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 01:20:24.714057 3900 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 01:20:24.714152 3900 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 01:20:24.714262 3900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:20:24.714370 3900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:24.714516 3900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:24.714614 3900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:20:24.714702 3900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:20:24.725075 3900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:34573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:34573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:24.726383 3900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:24.727844 3900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:24.740877 3905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:24.741070 3908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:24.740935 3906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:24.741890 3900 server_base.cc:1034] running on GCE node
I20260925 01:20:24.742865 3900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:24.744390 3900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:24.745668 3900 hybrid_clock.cc:648] HybridClock initialized: now 1790299224745575 us; error 63 us; skew 500 ppm
I20260925 01:20:24.746047 3900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:24 dist-test-slave-s12s krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299224, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:24.749379 3900 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:20:24.750952 3900 webserver.cc:460] Webserver started at http://127.20.102.190:40903/ using document root <none> and password file <none>
I20260925 01:20:24.751688 3900 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:24.751816 3900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:24.752135 3900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:20:24.755326 3900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "d5b3336265ff45a5bd6d02a5d4362080"
format_stamp: "Formatted at 2026-09-25 01:20:24 on dist-test-slave-s12s"
server_key: "0e2cf53bdc27346be784a4c63deb769a"
server_key_iv: "7711f02f98c41c24d4922eb95e86e3c7"
server_key_version: "encryptionkey@0"
I20260925 01:20:24.756122 3900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "d5b3336265ff45a5bd6d02a5d4362080"
format_stamp: "Formatted at 2026-09-25 01:20:24 on dist-test-slave-s12s"
server_key: "0e2cf53bdc27346be784a4c63deb769a"
server_key_iv: "7711f02f98c41c24d4922eb95e86e3c7"
server_key_version: "encryptionkey@0"
I20260925 01:20:24.761564 3900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260925 01:20:24.765286 3915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:24.766731 3900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:20:24.766912 3900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "d5b3336265ff45a5bd6d02a5d4362080"
format_stamp: "Formatted at 2026-09-25 01:20:24 on dist-test-slave-s12s"
server_key: "0e2cf53bdc27346be784a4c63deb769a"
server_key_iv: "7711f02f98c41c24d4922eb95e86e3c7"
server_key_version: "encryptionkey@0"
I20260925 01:20:24.767150 3900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:24.797582 3900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:24.801942 3900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:24.802249 3900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:24.821857 3900 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:34573
I20260925 01:20:24.821882 3966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:34573 every 8 connection(s)
I20260925 01:20:24.823262 3900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:20:24.826821 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 3900
I20260925 01:20:24.827023 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:20:24.827122 3967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:20:24.827776 20890 external_mini_cluster.cc:1428] Setting key 2406df11f60d1e41cdae8eec17c15cb0
I20260925 01:20:24.841858 3967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Bootstrap starting.
Sep 25 01:20:24 dist-test-slave-s12s krb5kdc[3887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:20:24.845861 3967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Neither blocks nor log segments found. Creating new log.
I20260925 01:20:24.847321 3967 log.cc:824] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:24.850883 3967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: No bootstrap required, opened a new log
I20260925 01:20:24.853104 3970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:24.829845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.20.102.190:34573)
0925 01:20:24.830698 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:24.830716 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:24.830765 (+ 49us) server_negotiation.cc:407] Connection header received
0925 01:20:24.832454 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:24.832487 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:24.833037 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:24.833507 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:24.834891 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:24.836840 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:24.838198 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:24.839312 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:24.839514 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:24.843958 (+ 4444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:24.843969 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:24.843995 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:24.844055 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:24.848194 (+ 4139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:24.849138 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:24.849146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:24.849155 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:24.849362 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:24.849807 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:24.849812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:24.849814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:24.850540 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:24.850837 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:24.851232 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 01:20:24.851721 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":348,"threads_started":1}
I20260925 01:20:24.859812 3967 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } }
I20260925 01:20:24.860172 3967 raft_consensus.cc:374] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:20:24.860244 3967 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b3336265ff45a5bd6d02a5d4362080, State: Initialized, Role: FOLLOWER
I20260925 01:20:24.860850 3967 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } }
I20260925 01:20:24.861111 3967 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:20:24.861253 3967 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:20:24.861416 3967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:20:24.863821 3967 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } }
I20260925 01:20:24.864316 3967 leader_election.cc:302] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5b3336265ff45a5bd6d02a5d4362080; no voters:
I20260925 01:20:24.865022 3967 leader_election.cc:288] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:20:24.865155 3972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:20:24.865859 3972 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 1 LEADER]: Becoming Leader. State: Replica: d5b3336265ff45a5bd6d02a5d4362080, State: Running, Role: LEADER
I20260925 01:20:24.866376 3972 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } }
I20260925 01:20:24.867118 3967 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:20:24.869387 3973 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5b3336265ff45a5bd6d02a5d4362080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } } }
I20260925 01:20:24.869359 3974 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5b3336265ff45a5bd6d02a5d4362080. Latest consensus state: current_term: 1 leader_uuid: "d5b3336265ff45a5bd6d02a5d4362080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } } }
I20260925 01:20:24.870229 3974 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: This master's current role is: LEADER
I20260925 01:20:24.871066 3973 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: This master's current role is: LEADER
I20260925 01:20:24.872117 3980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:20:24.876910 3980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:20:24.886183 3980 catalog_manager.cc:1347] Generated new cluster ID: d798082d39e846d2b073f16a607e911e
I20260925 01:20:24.886324 3980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:20:24.898666 3980 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:20:24.900220 3980 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:20:24.906849 3980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Generated new TSK 0
I20260925 01:20:24.907788 3980 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:20:24.973449 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:34573
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:20:25.309641 3993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:25.309945 3993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:25.310111 3993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:25.317317 3993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:25.317461 3993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:25.317628 3993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:20:25.327230 3993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:34573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:25.328538 3993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:25.329912 3993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:25.343472 3999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:25.343642 4001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:25.343688 3998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:25.349915 3993 server_base.cc:1034] running on GCE node
I20260925 01:20:25.350574 3993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:25.351665 3993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:25.352905 3993 hybrid_clock.cc:648] HybridClock initialized: now 1790299225352814 us; error 93 us; skew 500 ppm
I20260925 01:20:25.353251 3993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:25 dist-test-slave-s12s krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299225, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:25.356637 3993 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:20:25.358325 3993 webserver.cc:460] Webserver started at http://127.20.102.129:43929/ using document root <none> and password file <none>
I20260925 01:20:25.359050 3993 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:25.359191 3993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:25.359501 3993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:20:25.362501 3993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd"
format_stamp: "Formatted at 2026-09-25 01:20:25 on dist-test-slave-s12s"
server_key: "a3df6ab232db08252541af711e7bd9de"
server_key_iv: "365a509c3422c33f6ec90af5d2fb94bd"
server_key_version: "encryptionkey@0"
I20260925 01:20:25.363265 3993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd"
format_stamp: "Formatted at 2026-09-25 01:20:25 on dist-test-slave-s12s"
server_key: "a3df6ab232db08252541af711e7bd9de"
server_key_iv: "365a509c3422c33f6ec90af5d2fb94bd"
server_key_version: "encryptionkey@0"
I20260925 01:20:25.368774 3993 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:20:25.372517 4008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:25.373991 3993 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:20:25.374169 3993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd"
format_stamp: "Formatted at 2026-09-25 01:20:25 on dist-test-slave-s12s"
server_key: "a3df6ab232db08252541af711e7bd9de"
server_key_iv: "365a509c3422c33f6ec90af5d2fb94bd"
server_key_version: "encryptionkey@0"
I20260925 01:20:25.374372 3993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:25.394409 3993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:25.398886 3993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:25.399195 3993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:25.400137 3993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:25.401623 3993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:20:25.401722 3993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:25.401870 3993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:20:25.401938 3993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:25.436254 3993 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:39765
I20260925 01:20:25.436299 4120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:39765 every 8 connection(s)
I20260925 01:20:25.437618 3993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:20:25.438622 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 3993
I20260925 01:20:25.438846 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:20:25.439690 20890 external_mini_cluster.cc:1428] Setting key 89f5409818f1220f0f6b855b3451f3f4
Sep 25 01:20:25 dist-test-slave-s12s krb5kdc[3887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299225, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:20:25.459803 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:25.440648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:38613 (local address 127.20.102.190:34573)
0925 01:20:25.441179 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:25.441185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:25.442287 (+ 1102us) server_negotiation.cc:407] Connection header received
0925 01:20:25.444146 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:25.444151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:25.444275 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:25.444387 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:25.445937 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:25.446670 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:25.448214 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:25.448640 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:25.448797 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:25.452589 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:25.452597 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:25.452605 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:25.452650 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:25.456379 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:25.457162 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:25.457171 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:25.457177 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:25.457329 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:25.457750 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:25.457758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:25.457764 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:25.458167 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:25.458317 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:25.459069 (+ 752us) server_negotiation.cc:299] Negotiation successful
0925 01:20:25.459325 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":255,"threads_started":1}
I20260925 01:20:25.460634 4124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:25.441319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:38613)
0925 01:20:25.442151 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:25.442189 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:25.443887 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:25.444639 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:25.444647 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:25.445206 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:20:25.445773 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:25.445791 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:25.446824 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:25.446829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:25.448075 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:25.448087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:25.448927 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:25.448935 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:25.449283 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:25.450003 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:20:25.450033 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:20:25.452360 (+ 2327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:20:25.456525 (+ 4165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:25.456532 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:25.456544 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:25.457035 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:25.457437 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:25.457440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:25.457443 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:25.457630 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:25.458326 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:20:25.458332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:20:25.458667 (+ 335us) client_negotiation.cc:769] Sending connection context
0925 01:20:25.458888 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 01:20:25.459284 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":403,"threads_started":1}
I20260925 01:20:25.463162 4121 heartbeater.cc:347] Connected to a master server at 127.20.102.190:34573
I20260925 01:20:25.463531 4121 heartbeater.cc:464] Registering TS with master...
I20260925 01:20:25.464504 4121 heartbeater.cc:511] Master 127.20.102.190:34573 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
I20260925 01:20:25.467031 3932 ts_manager.cc:194] Registered new tserver with Master: d0f1c3572a5c41d4b81ad9fd41c69ccd (127.20.102.129:39765)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
I20260925 01:20:25.468775 3932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:38613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:20:25.497828 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:34573
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:20:25.828933 4127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:25.829236 4127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:25.829394 4127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:25.836845 4127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:25.836984 4127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:25.837160 4127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:20:25.847182 4127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:34573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:25.848492 4127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:25.849989 4127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:25.863737 4135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:25.863744 4133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:25.863668 4132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:25.864951 4127 server_base.cc:1034] running on GCE node
I20260925 01:20:25.865607 4127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:25.866643 4127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:25.867930 4127 hybrid_clock.cc:648] HybridClock initialized: now 1790299225867878 us; error 48 us; skew 500 ppm
I20260925 01:20:25.868222 4127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:25 dist-test-slave-s12s krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299225, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:25.871898 4127 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:20:25.873680 4127 webserver.cc:460] Webserver started at http://127.20.102.130:41457/ using document root <none> and password file <none>
I20260925 01:20:25.874383 4127 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:25.874514 4127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:25.874838 4127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:20:25.878016 4127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "2517a06cabfe4529868fccb0380b1073"
format_stamp: "Formatted at 2026-09-25 01:20:25 on dist-test-slave-s12s"
server_key: "416c21a236f550e3ab64553776960eac"
server_key_iv: "15fb1b40822197f0c76d69ec36769e68"
server_key_version: "encryptionkey@0"
I20260925 01:20:25.878882 4127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "2517a06cabfe4529868fccb0380b1073"
format_stamp: "Formatted at 2026-09-25 01:20:25 on dist-test-slave-s12s"
server_key: "416c21a236f550e3ab64553776960eac"
server_key_iv: "15fb1b40822197f0c76d69ec36769e68"
server_key_version: "encryptionkey@0"
I20260925 01:20:25.884261 4127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:20:25.889180 4142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:25.891525 4127 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260925 01:20:25.891745 4127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "2517a06cabfe4529868fccb0380b1073"
format_stamp: "Formatted at 2026-09-25 01:20:25 on dist-test-slave-s12s"
server_key: "416c21a236f550e3ab64553776960eac"
server_key_iv: "15fb1b40822197f0c76d69ec36769e68"
server_key_version: "encryptionkey@0"
I20260925 01:20:25.891999 4127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:25.909040 4127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:25.915174 4127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:25.915560 4127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:25.916963 4127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:25.918982 4127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:20:25.919101 4127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:25.919248 4127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:20:25.919323 4127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:25.953984 4127 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:44201
I20260925 01:20:25.954033 4254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:44201 every 8 connection(s)
I20260925 01:20:25.955366 4127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:20:25.962013 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 4127
I20260925 01:20:25.962229 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:20:25.962713 20890 external_mini_cluster.cc:1428] Setting key 6b460b881cdf7ac9814e7f1d5cbc2486
Sep 25 01:20:25 dist-test-slave-s12s krb5kdc[3887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299225, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:20:25.979580 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:25.957893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:54693 (local address 127.20.102.190:34573)
0925 01:20:25.958068 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:25.958072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:25.959638 (+ 1566us) server_negotiation.cc:407] Connection header received
0925 01:20:25.961623 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:25.961629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:25.961740 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:25.961868 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:25.963719 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:25.964838 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:25.967456 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:25.967917 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:25.968088 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:25.973428 (+ 5340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:25.973434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:25.973451 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:25.973489 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:25.976157 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:25.977038 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:25.977044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:25.977052 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:25.977190 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:25.977698 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:25.977702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:25.977704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:25.978037 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:25.978194 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:25.978955 (+ 761us) server_negotiation.cc:299] Negotiation successful
0925 01:20:25.979230 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:20:25.980639 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:25.958566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:54693)
0925 01:20:25.959464 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:25.959508 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:25.961332 (+ 1824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:25.962133 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:25.962141 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:25.962746 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:20:25.963531 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:25.963568 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:25.965025 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:25.965031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:25.967298 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:25.967323 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:25.968159 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:25.968169 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:25.968500 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:25.969517 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:20:25.969561 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:20:25.973133 (+ 3572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:20:25.976325 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:25.976335 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:25.976354 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:25.976910 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:25.977308 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:25.977313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:25.977317 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:25.977539 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:25.978168 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:20:25.978176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:20:25.978556 (+ 380us) client_negotiation.cc:769] Sending connection context
0925 01:20:25.978844 (+ 288us) client_negotiation.cc:240] Negotiation successful
0925 01:20:25.979460 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":284,"threads_started":1}
I20260925 01:20:25.982323 4255 heartbeater.cc:347] Connected to a master server at 127.20.102.190:34573
I20260925 01:20:25.982694 4255 heartbeater.cc:464] Registering TS with master...
I20260925 01:20:25.983619 4255 heartbeater.cc:511] Master 127.20.102.190:34573 requested a full tablet report, sending...
I20260925 01:20:25.985520 3932 ts_manager.cc:194] Registered new tserver with Master: 2517a06cabfe4529868fccb0380b1073 (127.20.102.130:44201)
I20260925 01:20:25.986814 3932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:54693
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:20:26.021816 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:34573
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:20:26.351423 4260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:26.351743 4260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:26.351907 4260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:26.359194 4260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:26.359310 4260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:26.359467 4260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:20:26.369342 4260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:34573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:26.370656 4260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:26.371980 4260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:26.384994 4266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:26.385020 4268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:26.385577 4265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:26.385677 4260 server_base.cc:1034] running on GCE node
I20260925 01:20:26.386336 4260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:26.387250 4260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:26.388473 4260 hybrid_clock.cc:648] HybridClock initialized: now 1790299226388412 us; error 53 us; skew 500 ppm
I20260925 01:20:26.388841 4260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:26 dist-test-slave-s12s krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299226, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:26.391925 4260 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:20:26.393491 4260 webserver.cc:460] Webserver started at http://127.20.102.131:42703/ using document root <none> and password file <none>
I20260925 01:20:26.394243 4260 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:26.394400 4260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:26.394728 4260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:20:26.397985 4260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c"
format_stamp: "Formatted at 2026-09-25 01:20:26 on dist-test-slave-s12s"
server_key: "7eb50acc54e1cea8cf8ae567abbe6447"
server_key_iv: "c752b6fe65436b88ef862506b27077a6"
server_key_version: "encryptionkey@0"
I20260925 01:20:26.398854 4260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c"
format_stamp: "Formatted at 2026-09-25 01:20:26 on dist-test-slave-s12s"
server_key: "7eb50acc54e1cea8cf8ae567abbe6447"
server_key_iv: "c752b6fe65436b88ef862506b27077a6"
server_key_version: "encryptionkey@0"
I20260925 01:20:26.404340 4260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 01:20:26.408140 4275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:26.409830 4260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:20:26.410029 4260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c"
format_stamp: "Formatted at 2026-09-25 01:20:26 on dist-test-slave-s12s"
server_key: "7eb50acc54e1cea8cf8ae567abbe6447"
server_key_iv: "c752b6fe65436b88ef862506b27077a6"
server_key_version: "encryptionkey@0"
I20260925 01:20:26.410264 4260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:26.424839 4260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:26.428849 4260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:26.429083 4260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:26.430015 4260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:26.431272 4260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:20:26.431346 4260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:26.431434 4260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:20:26.431481 4260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:26.471542 4260 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:46813
I20260925 01:20:26.471650 4387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:46813 every 8 connection(s)
I20260925 01:20:26.472003 4121 heartbeater.cc:503] Master 127.20.102.190:34573 was elected leader, sending a full tablet report...
I20260925 01:20:26.473069 4260 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:20:26.475448 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 4260
I20260925 01:20:26.475651 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:20:26.476289 20890 external_mini_cluster.cc:1428] Setting key 549f20e67ecbe482e5a0cf4d81944e6d
Sep 25 01:20:26 dist-test-slave-s12s krb5kdc[3887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299226, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:20:26.494095 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.476956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:47583 (local address 127.20.102.190:34573)
0925 01:20:26.477394 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:26.477399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:26.477746 (+ 347us) server_negotiation.cc:407] Connection header received
0925 01:20:26.480082 (+ 2336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:26.480086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:26.480174 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:26.480254 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:26.481823 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.482867 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.484398 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.484894 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.484994 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.488242 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:26.488248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:26.488251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:26.488283 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:26.491092 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:26.491793 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:26.491803 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:26.491806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:26.491930 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:26.492357 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:26.492363 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:26.492366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:26.492642 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:26.492828 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:26.493503 (+ 675us) server_negotiation.cc:299] Negotiation successful
0925 01:20:26.493737 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260925 01:20:26.495093 4390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.476620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:47583)
0925 01:20:26.477570 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:26.477633 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:26.479845 (+ 2212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:26.480462 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:26.480470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:26.481075 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:20:26.481642 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:26.481659 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.483008 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.483012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:26.484262 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:26.484274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.485090 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.485095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:26.485289 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.485907 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:20:26.485928 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:20:26.488059 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:20:26.491239 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:26.491248 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:26.491267 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:26.491667 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:26.492048 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:26.492057 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:26.492059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:26.492244 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:26.492794 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:20:26.492802 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:20:26.493145 (+ 343us) client_negotiation.cc:769] Sending connection context
0925 01:20:26.493406 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 01:20:26.493932 (+ 526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":366,"threads_started":1}
I20260925 01:20:26.496462 4388 heartbeater.cc:347] Connected to a master server at 127.20.102.190:34573
I20260925 01:20:26.496821 4388 heartbeater.cc:464] Registering TS with master...
I20260925 01:20:26.497622 4388 heartbeater.cc:511] Master 127.20.102.190:34573 requested a full tablet report, sending...
I20260925 01:20:26.499096 3932 ts_manager.cc:194] Registered new tserver with Master: 3eed8c8ed57f4bd5b74e428ad371dc1c (127.20.102.131:46813)
I20260925 01:20:26.499879 3932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:47583
I20260925 01:20:26.504078 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:20:26.521698 4123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.508661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41338 (local address 127.20.102.190:34573)
0925 01:20:26.508839 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:26.508844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:26.509405 (+ 561us) server_negotiation.cc:407] Connection header received
0925 01:20:26.509541 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:26.509545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:26.509607 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:26.509690 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:26.511159 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.511865 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.513459 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.513912 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.514005 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.516571 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:26.516576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:26.516579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:26.516614 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:26.519250 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:26.519938 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:26.519942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:26.519944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:26.520096 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:26.520432 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:26.520437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:26.520440 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:26.520839 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:26.521040 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:26.521292 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 01:20:26.521458 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:20:26.525914 3932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41338:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:20:26.533581 3932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:20:26.557264 4400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.546575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:46813 (local address 127.0.0.1:40848)
0925 01:20:26.547411 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:26.547451 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:26.547605 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:26.548795 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:26.548806 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:26.548850 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:26.549369 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:26.549382 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.551588 (+ 2206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.551593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:26.554129 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:26.554146 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.556184 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.556192 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:26.556377 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.556435 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 01:20:26.556565 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 01:20:26.556811 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":1,"thread_start_us":430,"threads_started":1}
I20260925 01:20:26.557502 4401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.546739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40848 (local address 127.20.102.131:46813)
0925 01:20:26.547529 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:26.547537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:26.547568 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:20:26.547860 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:26.547871 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:26.548187 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:26.548418 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:26.549597 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.551381 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.554327 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.555974 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.556198 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.556338 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:26.556973 (+ 635us) server_negotiation.cc:299] Negotiation successful
0925 01:20:26.557149 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":473,"threads_started":1}
I20260925 01:20:26.558845 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.548451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39765 (local address 127.0.0.1:54058)
0925 01:20:26.549218 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:26.549242 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:26.549372 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:26.550642 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:26.550648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:26.550678 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:26.551279 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:26.551290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.553273 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.553278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:26.555201 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:26.555213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.556700 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.556708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:26.558190 (+ 1482us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.558225 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:20:26.558321 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 01:20:26.558447 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"mutex_wait_us":128,"thread_start_us":689,"threads_started":1}
I20260925 01:20:26.559106 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.549015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.20.102.129:39765)
0925 01:20:26.550076 (+ 1061us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:26.550089 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:26.550120 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:20:26.550215 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:26.550223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:26.550442 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:26.550633 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:26.551471 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.553114 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.555360 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.556437 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.556651 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.556851 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:26.558523 (+ 1672us) server_negotiation.cc:299] Negotiation successful
0925 01:20:26.558732 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":862,"thread_start_us":687,"threads_started":1}
I20260925 01:20:26.559727 4402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.547827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:44201 (local address 127.0.0.1:38422)
0925 01:20:26.548414 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:26.548439 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:26.548603 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:26.550785 (+ 2182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:26.550790 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:26.550815 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:26.551441 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:26.551459 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.553338 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.553347 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:26.555954 (+ 2607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:26.555970 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.558254 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.558261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:26.558454 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.558491 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 01:20:26.558597 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 01:20:26.558738 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":119,"thread_start_us":294,"threads_started":1}
I20260925 01:20:26.560591 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.548140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.20.102.130:44201)
0925 01:20:26.550273 (+ 2133us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:26.550279 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:26.550301 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:20:26.550374 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:26.550379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:26.550544 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:26.550992 (+ 448us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:26.551681 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.553155 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.556426 (+ 3271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.558020 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.558217 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.558402 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:26.560072 (+ 1670us) server_negotiation.cc:299] Negotiation successful
0925 01:20:26.560279 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1986,"thread_start_us":1888,"threads_started":1}
I20260925 01:20:26.562645 4056 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc47b897cda4b0c93742b1b68bce6a2 (DEFAULT_TABLE table=test-table [id=c00f5040fa0146a88024147a2839f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:20:26.562804 4323 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc47b897cda4b0c93742b1b68bce6a2 (DEFAULT_TABLE table=test-table [id=c00f5040fa0146a88024147a2839f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:20:26.564188 4323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc47b897cda4b0c93742b1b68bce6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:20:26.564205 4056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc47b897cda4b0c93742b1b68bce6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:20:26.564160 4190 tablet_service.cc:1467] Processing CreateTablet for tablet 3fc47b897cda4b0c93742b1b68bce6a2 (DEFAULT_TABLE table=test-table [id=c00f5040fa0146a88024147a2839f376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:20:26.565496 4190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fc47b897cda4b0c93742b1b68bce6a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:20:26.577843 4406 tablet_bootstrap.cc:492] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Bootstrap starting.
I20260925 01:20:26.578886 4407 tablet_bootstrap.cc:492] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Bootstrap starting.
I20260925 01:20:26.581391 4408 tablet_bootstrap.cc:492] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073: Bootstrap starting.
I20260925 01:20:26.582145 4406 tablet_bootstrap.cc:654] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Neither blocks nor log segments found. Creating new log.
I20260925 01:20:26.582926 4407 tablet_bootstrap.cc:654] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Neither blocks nor log segments found. Creating new log.
I20260925 01:20:26.583599 4406 log.cc:824] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:26.584231 4408 tablet_bootstrap.cc:654] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073: Neither blocks nor log segments found. Creating new log.
I20260925 01:20:26.584372 4407 log.cc:824] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:26.585418 4408 log.cc:824] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:26.588054 4407 tablet_bootstrap.cc:492] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd: No bootstrap required, opened a new log
I20260925 01:20:26.588361 4407 ts_tablet_manager.cc:1397] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 01:20:26.588915 4406 tablet_bootstrap.cc:492] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c: No bootstrap required, opened a new log
I20260925 01:20:26.589197 4406 ts_tablet_manager.cc:1397] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20260925 01:20:26.591408 4408 tablet_bootstrap.cc:492] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073: No bootstrap required, opened a new log
I20260925 01:20:26.591667 4408 ts_tablet_manager.cc:1397] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 01:20:26.597364 4407 raft_consensus.cc:348] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } } peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } }
I20260925 01:20:26.597829 4407 raft_consensus.cc:374] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:20:26.597940 4407 raft_consensus.cc:729] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f1c3572a5c41d4b81ad9fd41c69ccd, State: Initialized, Role: FOLLOWER
I20260925 01:20:26.598616 4407 consensus_queue.cc:260] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } } peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } }
I20260925 01:20:26.599859 4406 raft_consensus.cc:348] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } } peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } }
I20260925 01:20:26.600308 4406 raft_consensus.cc:374] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:20:26.600410 4406 raft_consensus.cc:729] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eed8c8ed57f4bd5b74e428ad371dc1c, State: Initialized, Role: FOLLOWER
I20260925 01:20:26.600543 4407 ts_tablet_manager.cc:1428] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Time spent starting tablet: real 0.012s user 0.003s sys 0.010s
I20260925 01:20:26.601012 4406 consensus_queue.cc:260] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } } peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } }
I20260925 01:20:26.604166 4388 heartbeater.cc:503] Master 127.20.102.190:34573 was elected leader, sending a full tablet report...
I20260925 01:20:26.604933 4406 ts_tablet_manager.cc:1428] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260925 01:20:26.604996 4408 raft_consensus.cc:348] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } } peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } }
I20260925 01:20:26.605508 4408 raft_consensus.cc:374] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:20:26.605609 4408 raft_consensus.cc:729] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2517a06cabfe4529868fccb0380b1073, State: Initialized, Role: FOLLOWER
I20260925 01:20:26.606235 4408 consensus_queue.cc:260] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } } peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } }
I20260925 01:20:26.607532 4255 heartbeater.cc:503] Master 127.20.102.190:34573 was elected leader, sending a full tablet report...
I20260925 01:20:26.607924 4408 ts_tablet_manager.cc:1428] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
W20260925 01:20:26.690186 4122 tablet.cc:2369] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:20:26.700533 4414 raft_consensus.cc:482] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:20:26.700861 4414 raft_consensus.cc:504] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } } peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } }
I20260925 01:20:26.702497 4414 leader_election.cc:288] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3eed8c8ed57f4bd5b74e428ad371dc1c (127.20.102.131:46813), d0f1c3572a5c41d4b81ad9fd41c69ccd (127.20.102.129:39765)
W20260925 01:20:26.707369 4256 tablet.cc:2369] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:20:26.709362 4401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.702902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:49407 (local address 127.20.102.131:46813)
0925 01:20:26.703175 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:26.703182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:26.703475 (+ 293us) server_negotiation.cc:407] Connection header received
0925 01:20:26.703733 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:26.703742 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:26.703836 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:26.703936 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:26.704514 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.705343 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.707318 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.708396 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.708571 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.708650 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:26.708968 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 01:20:26.709102 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:20:26.709507 4415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.702758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:46813 (local address 127.20.102.130:49407)
0925 01:20:26.703373 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:26.703389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:26.703588 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:26.703994 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:26.704001 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:26.704037 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:26.704364 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:26.704373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.705464 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.705471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:26.707142 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:26.707159 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.708609 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.708616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:26.708812 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.708836 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:20:26.708940 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 01:20:26.709066 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":19,"thread_start_us":329,"threads_started":1}
I20260925 01:20:26.709969 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.703646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:49601 (local address 127.20.102.129:39765)
0925 01:20:26.703891 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:26.703897 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:26.704263 (+ 366us) server_negotiation.cc:407] Connection header received
0925 01:20:26.704459 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:26.704463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:26.704522 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:26.704615 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:26.705172 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.705989 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.708101 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.708978 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.709148 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.709227 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:26.709595 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 01:20:26.709735 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 01:20:26.710695 4416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.703467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39765 (local address 127.20.102.130:49601)
0925 01:20:26.704157 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:26.704178 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:26.704317 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:26.704629 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:26.704633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:26.704655 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:26.705014 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:26.705022 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.706127 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.706132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:26.707927 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:26.707944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.709214 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.709222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:26.709383 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.709406 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 01:20:26.710185 (+ 779us) client_negotiation.cc:240] Negotiation successful
0925 01:20:26.710320 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":3,"thread_start_us":366,"threads_started":1}
I20260925 01:20:26.710624 4343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc47b897cda4b0c93742b1b68bce6a2" candidate_uuid: "2517a06cabfe4529868fccb0380b1073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" is_pre_election: true
I20260925 01:20:26.711222 4343 raft_consensus.cc:2454] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2517a06cabfe4529868fccb0380b1073 in term 0.
I20260925 01:20:26.711700 4076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc47b897cda4b0c93742b1b68bce6a2" candidate_uuid: "2517a06cabfe4529868fccb0380b1073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" is_pre_election: true
I20260925 01:20:26.711903 4146 leader_election.cc:302] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2517a06cabfe4529868fccb0380b1073, 3eed8c8ed57f4bd5b74e428ad371dc1c; no voters:
I20260925 01:20:26.712327 4414 raft_consensus.cc:2781] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:20:26.712268 4076 raft_consensus.cc:2454] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2517a06cabfe4529868fccb0380b1073 in term 0.
I20260925 01:20:26.712477 4414 raft_consensus.cc:482] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:20:26.712611 4414 raft_consensus.cc:3037] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:20:26.714859 4414 raft_consensus.cc:504] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } } peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } }
I20260925 01:20:26.715454 4414 leader_election.cc:288] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [CANDIDATE]: Term 1 election: Requested vote from peers 3eed8c8ed57f4bd5b74e428ad371dc1c (127.20.102.131:46813), d0f1c3572a5c41d4b81ad9fd41c69ccd (127.20.102.129:39765)
I20260925 01:20:26.715886 4343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc47b897cda4b0c93742b1b68bce6a2" candidate_uuid: "2517a06cabfe4529868fccb0380b1073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c"
I20260925 01:20:26.715971 4076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fc47b897cda4b0c93742b1b68bce6a2" candidate_uuid: "2517a06cabfe4529868fccb0380b1073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd"
I20260925 01:20:26.716074 4343 raft_consensus.cc:3037] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:20:26.716207 4076 raft_consensus.cc:3037] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:20:26.718372 4343 raft_consensus.cc:2454] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2517a06cabfe4529868fccb0380b1073 in term 1.
I20260925 01:20:26.718480 4076 raft_consensus.cc:2454] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2517a06cabfe4529868fccb0380b1073 in term 1.
I20260925 01:20:26.718770 4146 leader_election.cc:302] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2517a06cabfe4529868fccb0380b1073, 3eed8c8ed57f4bd5b74e428ad371dc1c; no voters:
I20260925 01:20:26.719022 4414 raft_consensus.cc:2781] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:20:26.719465 4414 raft_consensus.cc:686] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [term 1 LEADER]: Becoming Leader. State: Replica: 2517a06cabfe4529868fccb0380b1073, State: Running, Role: LEADER
I20260925 01:20:26.719949 4414 consensus_queue.cc:237] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } } peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } }
I20260925 01:20:26.724443 3932 catalog_manager.cc:5495] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2517a06cabfe4529868fccb0380b1073 (127.20.102.130). New cstate: current_term: 1 leader_uuid: "2517a06cabfe4529868fccb0380b1073" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } health_report { overall_health: UNKNOWN } } }
W20260925 01:20:26.724931 4389 tablet.cc:2369] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:20:26.750317 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.744794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38432 (local address 127.20.102.130:44201)
0925 01:20:26.744968 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:26.744972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:26.745014 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:20:26.745257 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:26.745261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:26.745342 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:26.745468 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:20:26.745861 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.746583 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.748050 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.748448 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.748550 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.749237 (+ 687us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:20:26.749775 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:20:26.749887 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:26.750022 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 01:20:26.750089 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 01:20:26.758548 4076 raft_consensus.cc:1261] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 1 FOLLOWER]: Refusing update from remote peer 2517a06cabfe4529868fccb0380b1073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:20:26.758642 4343 raft_consensus.cc:1261] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 1 FOLLOWER]: Refusing update from remote peer 2517a06cabfe4529868fccb0380b1073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:20:26.759523 4417 consensus_queue.cc:1035] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:20:26.759907 4414 consensus_queue.cc:1035] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:20:26.771473 4420 mvcc.cc:205] Tried to move back new op lower bound from 7333065632793399296 to 7333065632649220096. Current Snapshot: MvccSnapshot[applied={T|T < 7333065632793399296}]
I20260925 01:20:26.778205 4422 mvcc.cc:205] Tried to move back new op lower bound from 7333065632793399296 to 7333065632649220096. Current Snapshot: MvccSnapshot[applied={T|T < 7333065632793399296}]
I20260925 01:20:26.787092 3931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:20:26.787418 3931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41338:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:20:26.791356 3931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Sending DeleteTablet for 3 replicas of tablet 3fc47b897cda4b0c93742b1b68bce6a2
I20260925 01:20:26.792662 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 3993
I20260925 01:20:26.792644 4056 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc47b897cda4b0c93742b1b68bce6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:20:26 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:54058
I20260925 01:20:26.792830 4190 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc47b897cda4b0c93742b1b68bce6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:20:26 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:38422
I20260925 01:20:26.793529 4323 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fc47b897cda4b0c93742b1b68bce6a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:20:26 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:40848
I20260925 01:20:26.804554 4432 tablet_replica.cc:330] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073: stopping tablet replica
I20260925 01:20:26.806668 4432 raft_consensus.cc:2229] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:20:26.807900 4432 raft_consensus.cc:2258] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:20:26.807962 4431 tablet_replica.cc:330] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd: stopping tablet replica
I20260925 01:20:26.808367 4434 tablet_replica.cc:330] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c: stopping tablet replica
I20260925 01:20:26.808411 4431 raft_consensus.cc:2229] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:20:26.809017 4434 raft_consensus.cc:2229] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:20:26.809135 4431 raft_consensus.cc:2258] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:20:26.809659 4434 raft_consensus.cc:2258] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:20:26.811400 4432 ts_tablet_manager.cc:1905] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:20:26.811510 4431 ts_tablet_manager.cc:1905] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:20:26.812000 4434 ts_tablet_manager.cc:1905] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:20:26.818955 4431 ts_tablet_manager.cc:1918] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:20:26.818990 4432 ts_tablet_manager.cc:1918] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:20:26.819175 4431 log.cc:1196] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/3fc47b897cda4b0c93742b1b68bce6a2
I20260925 01:20:26.819211 4432 log.cc:1196] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/3fc47b897cda4b0c93742b1b68bce6a2
I20260925 01:20:26.819777 4431 ts_tablet_manager.cc:1939] T 3fc47b897cda4b0c93742b1b68bce6a2 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Deleting consensus metadata
I20260925 01:20:26.819957 4432 ts_tablet_manager.cc:1939] T 3fc47b897cda4b0c93742b1b68bce6a2 P 2517a06cabfe4529868fccb0380b1073: Deleting consensus metadata
I20260925 01:20:26.820865 4434 ts_tablet_manager.cc:1918] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:20:26.821045 4434 log.cc:1196] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/3fc47b897cda4b0c93742b1b68bce6a2
I20260925 01:20:26.821834 3919 catalog_manager.cc:4841] TS d0f1c3572a5c41d4b81ad9fd41c69ccd (127.20.102.129:39765): tablet 3fc47b897cda4b0c93742b1b68bce6a2 (table test-table [id=c00f5040fa0146a88024147a2839f376]) successfully deleted
I20260925 01:20:26.822206 3919 catalog_manager.cc:4841] TS 2517a06cabfe4529868fccb0380b1073 (127.20.102.130:44201): tablet 3fc47b897cda4b0c93742b1b68bce6a2 (table test-table [id=c00f5040fa0146a88024147a2839f376]) successfully deleted
Sep 25 01:20:26 dist-test-slave-s12s krb5kdc[3887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:20:26.824183 4434 ts_tablet_manager.cc:1939] T 3fc47b897cda4b0c93742b1b68bce6a2 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Deleting consensus metadata
I20260925 01:20:26.826910 3919 catalog_manager.cc:4841] TS 3eed8c8ed57f4bd5b74e428ad371dc1c (127.20.102.131:46813): tablet 3fc47b897cda4b0c93742b1b68bce6a2 (table test-table [id=c00f5040fa0146a88024147a2839f376]) successfully deleted
I20260925 01:20:26.830891 4405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.794328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.20.102.129:39765)
0925 01:20:26.794554 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:26.794563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:26.810805 (+ 16242us) server_negotiation.cc:407] Connection header received
0925 01:20:26.811151 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:26.811157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:26.811234 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:26.811318 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:26.813535 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.814681 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.816967 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.817675 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.817798 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.823791 (+ 5993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:26.823805 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:26.823823 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:26.823870 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:26.827648 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:26.828518 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:26.828528 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:26.828539 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:26.828737 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:26.829191 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:26.829196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:26.829199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:26.829643 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:26.829968 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:26.830301 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 01:20:26.830518 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 01:20:26.831559 4116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:26.954149 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 3993
I20260925 01:20:26.974638 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 4127
Sep 25 01:20:26 dist-test-slave-s12s krb5kdc[3887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:20:26.991791 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:26.975460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38434 (local address 127.20.102.130:44201)
0925 01:20:26.975667 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:26.975672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:26.975911 (+ 239us) server_negotiation.cc:407] Connection header received
0925 01:20:26.976113 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:26.976119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:26.976215 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:26.976327 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:26.977677 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.978565 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.979910 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:26.980358 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:26.980491 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:26.985430 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:26.985447 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:26.985463 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:26.985501 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:26.988575 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:26.989333 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:26.989348 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:26.989351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:26.989493 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:26.989901 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:26.989905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:26.989907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:26.990287 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:26.990743 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:26.991126 (+ 383us) server_negotiation.cc:299] Negotiation successful
0925 01:20:26.991357 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:20:26.992576 4250 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:27.106767 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 4127
I20260925 01:20:27.126911 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 4260
Sep 25 01:20:27 dist-test-slave-s12s krb5kdc[3887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:20:27.142978 4401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:27.127739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40854 (local address 127.20.102.131:46813)
0925 01:20:27.127905 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:27.127910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:27.127933 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:20:27.128074 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:27.128078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:27.128142 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:27.128231 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:27.129632 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:27.130359 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:27.131707 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:27.132174 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:27.132320 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:27.137044 (+ 4724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:27.137060 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:27.137077 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:27.137122 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:27.139891 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:27.140607 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:27.140615 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:27.140617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:27.140903 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:27.141312 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:27.141352 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:27.141355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:27.141729 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:27.142040 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:27.142394 (+ 354us) server_negotiation.cc:299] Negotiation successful
0925 01:20:27.142635 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:20:27.143636 4383 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:27.231892 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 4260
I20260925 01:20:27.249320 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 3900
I20260925 01:20:27.250646 3962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:27.363294 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 3900
I20260925 01:20:47.385870 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:34573
--webserver_interface=127.20.102.190
--webserver_port=40903
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:34573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:20:47.730847 4439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:47.731160 4439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:47.731261 4439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:47.738607 4439 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 01:20:47.738710 4439 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 01:20:47.738750 4439 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 01:20:47.738801 4439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:20:47.738853 4439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:47.738914 4439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:47.738952 4439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:20:47.738999 4439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:20:47.748643 4439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:34573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:34573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=40903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:47.749979 4439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:47.751257 4439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:47.774760 4445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:47.774760 4446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:47.774785 4448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:47.778232 4439 server_base.cc:1034] running on GCE node
I20260925 01:20:47.779143 4439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:47.780841 4439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:47.782119 4439 hybrid_clock.cc:648] HybridClock initialized: now 1790299247782049 us; error 53 us; skew 500 ppm
I20260925 01:20:47.782562 4439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:47 dist-test-slave-s12s krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299247, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:47.786381 4439 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:20:47.788260 4439 webserver.cc:460] Webserver started at http://127.20.102.190:40903/ using document root <none> and password file <none>
I20260925 01:20:47.789219 4439 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:47.789383 4439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:47.795809 4439 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 01:20:47.799010 4455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:47.800740 4439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:20:47.800951 4439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "d5b3336265ff45a5bd6d02a5d4362080"
format_stamp: "Formatted at 2026-09-25 01:20:24 on dist-test-slave-s12s"
server_key: "0e2cf53bdc27346be784a4c63deb769a"
server_key_iv: "7711f02f98c41c24d4922eb95e86e3c7"
server_key_version: "encryptionkey@0"
I20260925 01:20:47.802044 4439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:47.836381 4439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:47.840894 4439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:47.841233 4439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:47.861202 4439 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:34573
I20260925 01:20:47.861245 4506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:34573 every 8 connection(s)
I20260925 01:20:47.862865 4439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:20:47.870085 4507 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:20:47.872238 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 4439
I20260925 01:20:47.873322 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:39765
--local_ip_for_outbound_sockets=127.20.102.129
--tserver_master_addrs=127.20.102.190:34573
--webserver_port=43929
--webserver_interface=127.20.102.129
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:20:47.874279 4507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Bootstrap starting.
I20260925 01:20:47.896871 4507 log.cc:824] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:47.910126 4507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:20:47.910727 4507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Bootstrap complete.
I20260925 01:20:47.920511 4507 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } }
I20260925 01:20:47.921437 4507 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b3336265ff45a5bd6d02a5d4362080, State: Initialized, Role: FOLLOWER
I20260925 01:20:47.922106 4507 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } }
I20260925 01:20:47.922355 4507 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:20:47.922483 4507 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:20:47.922674 4507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 1 FOLLOWER]: Advancing to term 2
I20260925 01:20:47.926831 4507 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } }
I20260925 01:20:47.927417 4507 leader_election.cc:302] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5b3336265ff45a5bd6d02a5d4362080; no voters:
I20260925 01:20:47.928128 4507 leader_election.cc:288] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 01:20:47.928284 4511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 01:20:47.929759 4511 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 2 LEADER]: Becoming Leader. State: Replica: d5b3336265ff45a5bd6d02a5d4362080, State: Running, Role: LEADER
I20260925 01:20:47.930415 4511 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } }
I20260925 01:20:47.930685 4507 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:20:47.933208 4513 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5b3336265ff45a5bd6d02a5d4362080. Latest consensus state: current_term: 2 leader_uuid: "d5b3336265ff45a5bd6d02a5d4362080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } } }
I20260925 01:20:47.933600 4513 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: This master's current role is: LEADER
I20260925 01:20:47.934175 4512 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d5b3336265ff45a5bd6d02a5d4362080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } } }
I20260925 01:20:47.934465 4512 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: This master's current role is: LEADER
I20260925 01:20:47.942157 4518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:20:47.950399 4518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:20:47.951884 4518 catalog_manager.cc:1259] Loaded cluster ID: d798082d39e846d2b073f16a607e911e
I20260925 01:20:47.952040 4518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:20:47.956233 4518 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:20:47.958094 4518 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Loaded TSK: 0
I20260925 01:20:47.992748 4518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Generated new TSK 1
I20260925 01:20:47.993284 4518 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 01:20:48.242545 4509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:48.242889 4509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:48.243083 4509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:48.250471 4509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:48.250581 4509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:48.250754 4509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:20:48.260571 4509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:39765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=43929
--tserver_master_addrs=127.20.102.190:34573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:48.261986 4509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:48.263355 4509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:48.277393 4535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:48.277557 4537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:48.278131 4534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:48.278795 4509 server_base.cc:1034] running on GCE node
I20260925 01:20:48.279603 4509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:48.280923 4509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:48.282207 4509 hybrid_clock.cc:648] HybridClock initialized: now 1790299248282130 us; error 62 us; skew 500 ppm
I20260925 01:20:48.282585 4509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:48 dist-test-slave-s12s krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299248, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:48.286757 4509 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:20:48.288652 4509 webserver.cc:460] Webserver started at http://127.20.102.129:43929/ using document root <none> and password file <none>
I20260925 01:20:48.289578 4509 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:48.289734 4509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:48.297433 4509 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 01:20:48.301083 4544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:48.302575 4509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:20:48.302763 4509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd"
format_stamp: "Formatted at 2026-09-25 01:20:25 on dist-test-slave-s12s"
server_key: "a3df6ab232db08252541af711e7bd9de"
server_key_iv: "365a509c3422c33f6ec90af5d2fb94bd"
server_key_version: "encryptionkey@0"
I20260925 01:20:48.303797 4509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:48.321298 4509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:48.325970 4509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:48.326265 4509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:48.327291 4509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:48.328624 4509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:20:48.328773 4509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:48.328862 4509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:20:48.328922 4509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:48.362663 4509 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:39765
I20260925 01:20:48.362761 4656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:39765 every 8 connection(s)
I20260925 01:20:48.364217 4509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:20:48.371757 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 4509
I20260925 01:20:48.372447 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:44201
--local_ip_for_outbound_sockets=127.20.102.130
--tserver_master_addrs=127.20.102.190:34573
--webserver_port=41457
--webserver_interface=127.20.102.130
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:20:48 dist-test-slave-s12s krb5kdc[3887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299248, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:20:48.388556 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:48.367290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:47293 (local address 127.20.102.190:34573)
0925 01:20:48.367965 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:48.367986 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:48.368600 (+ 614us) server_negotiation.cc:407] Connection header received
0925 01:20:48.370742 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:48.370755 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:48.371119 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:48.371504 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:48.373143 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:48.374403 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:48.375992 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:48.376648 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:48.376920 (+ 272us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:48.380957 (+ 4037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:48.380965 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:48.380979 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:48.381019 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:48.384019 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:48.385026 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:48.385033 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:48.385038 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:48.385207 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:48.385698 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:48.385702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:48.385703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:48.386150 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:48.386300 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:48.387270 (+ 970us) server_negotiation.cc:299] Negotiation successful
0925 01:20:48.387601 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":246,"threads_started":1}
I20260925 01:20:48.389176 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:48.367563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:47293)
0925 01:20:48.368377 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:48.368415 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:48.370193 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:48.371664 (+ 1471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:48.371677 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:48.372314 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:20:48.372941 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:48.372959 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:48.374580 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:48.374588 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:48.375832 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:48.375844 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:48.376939 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:48.376947 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:48.377248 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:48.377940 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:20:48.377963 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:20:48.380658 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:20:48.384176 (+ 3518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:48.384188 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:48.384208 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:48.384878 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:48.385326 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:48.385333 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:48.385340 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:48.385585 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:48.386290 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:20:48.386301 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:20:48.386766 (+ 465us) client_negotiation.cc:769] Sending connection context
0925 01:20:48.387063 (+ 297us) client_negotiation.cc:240] Negotiation successful
0925 01:20:48.387635 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":316,"threads_started":1}
I20260925 01:20:48.392294 4657 heartbeater.cc:347] Connected to a master server at 127.20.102.190:34573
I20260925 01:20:48.392650 4657 heartbeater.cc:464] Registering TS with master...
I20260925 01:20:48.393687 4657 heartbeater.cc:511] Master 127.20.102.190:34573 requested a full tablet report, sending...
I20260925 01:20:48.396392 4472 ts_manager.cc:194] Registered new tserver with Master: d0f1c3572a5c41d4b81ad9fd41c69ccd (127.20.102.129:39765)
I20260925 01:20:48.398010 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:47293
W20260925 01:20:48.708858 4661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:48.709122 4661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:48.709291 4661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:48.716336 4661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:48.716440 4661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:48.716691 4661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:20:48.726186 4661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:44201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=41457
--tserver_master_addrs=127.20.102.190:34573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:48.727430 4661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:48.728852 4661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:48.742009 4667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:48.742208 4666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:48.742060 4669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:48.743263 4661 server_base.cc:1034] running on GCE node
I20260925 01:20:48.743939 4661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:48.744899 4661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:48.746167 4661 hybrid_clock.cc:648] HybridClock initialized: now 1790299248746128 us; error 50 us; skew 500 ppm
I20260925 01:20:48.746551 4661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:48 dist-test-slave-s12s krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299248, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:48.749796 4661 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:20:48.751220 4661 webserver.cc:460] Webserver started at http://127.20.102.130:41457/ using document root <none> and password file <none>
I20260925 01:20:48.751899 4661 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:48.752060 4661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:48.757477 4661 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:20:48.760151 4676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:48.761524 4661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:20:48.761708 4661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "2517a06cabfe4529868fccb0380b1073"
format_stamp: "Formatted at 2026-09-25 01:20:25 on dist-test-slave-s12s"
server_key: "416c21a236f550e3ab64553776960eac"
server_key_iv: "15fb1b40822197f0c76d69ec36769e68"
server_key_version: "encryptionkey@0"
I20260925 01:20:48.762643 4661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:48.777408 4661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:48.781404 4661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:48.781706 4661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:48.782685 4661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:48.784049 4661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:20:48.784192 4661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:48.784330 4661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:20:48.784421 4661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:48.819386 4661 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:44201
I20260925 01:20:48.819439 4788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:44201 every 8 connection(s)
I20260925 01:20:48.821015 4661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:20:48.828717 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 4661
I20260925 01:20:48.829358 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:46813
--local_ip_for_outbound_sockets=127.20.102.131
--tserver_master_addrs=127.20.102.190:34573
--webserver_port=42703
--webserver_interface=127.20.102.131
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 01:20:48 dist-test-slave-s12s krb5kdc[3887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299248, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:20:48.843387 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:48.823582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:55481 (local address 127.20.102.190:34573)
0925 01:20:48.823761 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:48.823765 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:48.824935 (+ 1170us) server_negotiation.cc:407] Connection header received
0925 01:20:48.826744 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:48.826750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:48.826844 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:48.826955 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:48.828483 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:48.829339 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:48.831890 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:48.832454 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:48.832587 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:48.837202 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:48.837212 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:48.837219 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:48.837266 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:48.839996 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:48.840933 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:48.840937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:48.840940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:48.841058 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:48.841555 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:48.841558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:48.841560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:48.841898 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:48.842011 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:48.842886 (+ 875us) server_negotiation.cc:299] Negotiation successful
0925 01:20:48.843065 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:20:48.844887 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:48.824040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:55481)
0925 01:20:48.824780 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:48.824826 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:48.826487 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:48.827194 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:48.827202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:48.827692 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:20:48.828286 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:48.828312 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:48.829511 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:48.829518 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:48.831750 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:48.831769 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:48.832706 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:48.832716 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:48.833032 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:48.833817 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:20:48.833871 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:20:48.836939 (+ 3068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:20:48.840147 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:48.840159 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:48.840179 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:48.840819 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:48.841173 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:48.841180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:48.841187 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:48.841430 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:48.842037 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:20:48.842048 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:20:48.842482 (+ 434us) client_negotiation.cc:769] Sending connection context
0925 01:20:48.842799 (+ 317us) client_negotiation.cc:240] Negotiation successful
0925 01:20:48.843371 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":261,"threads_started":1}
I20260925 01:20:48.846486 4789 heartbeater.cc:347] Connected to a master server at 127.20.102.190:34573
I20260925 01:20:48.846808 4789 heartbeater.cc:464] Registering TS with master...
I20260925 01:20:48.847553 4789 heartbeater.cc:511] Master 127.20.102.190:34573 requested a full tablet report, sending...
I20260925 01:20:48.849151 4472 ts_manager.cc:194] Registered new tserver with Master: 2517a06cabfe4529868fccb0380b1073 (127.20.102.130:44201)
I20260925 01:20:48.850457 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:55481
W20260925 01:20:49.169891 4792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:20:49.170161 4792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:20:49.170325 4792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:20:49.177590 4792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:20:49.177693 4792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:20:49.177917 4792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:20:49.187366 4792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:46813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=42703
--tserver_master_addrs=127.20.102.190:34573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:20:49.188694 4792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:20:49.190001 4792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:20:49.204300 4798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:49.204254 4797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:20:49.204773 4800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:20:49.205262 4792 server_base.cc:1034] running on GCE node
I20260925 01:20:49.206122 4792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:20:49.207223 4792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:20:49.208434 4792 hybrid_clock.cc:648] HybridClock initialized: now 1790299249208388 us; error 40 us; skew 500 ppm
I20260925 01:20:49.208866 4792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:20:49 dist-test-slave-s12s krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299249, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:20:49.212193 4792 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:20:49.213743 4792 webserver.cc:460] Webserver started at http://127.20.102.131:42703/ using document root <none> and password file <none>
I20260925 01:20:49.214453 4792 fs_manager.cc:359] Metadata directory not provided
I20260925 01:20:49.214625 4792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:20:49.220489 4792 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 01:20:49.223341 4807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:49.224785 4792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:20:49.224958 4792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c"
format_stamp: "Formatted at 2026-09-25 01:20:26 on dist-test-slave-s12s"
server_key: "7eb50acc54e1cea8cf8ae567abbe6447"
server_key_iv: "c752b6fe65436b88ef862506b27077a6"
server_key_version: "encryptionkey@0"
I20260925 01:20:49.225909 4792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:20:49.243714 4792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:20:49.248458 4792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:20:49.248817 4792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:20:49.249867 4792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:20:49.251252 4792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:20:49.251399 4792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:49.251530 4792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:20:49.251631 4792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:20:49.288435 4792 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:46813
I20260925 01:20:49.288479 4919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:46813 every 8 connection(s)
I20260925 01:20:49.290094 4792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:20:49.297883 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 4792
Sep 25 01:20:49 dist-test-slave-s12s krb5kdc[3887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299249, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:20:49.312738 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.292944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:60235 (local address 127.20.102.190:34573)
0925 01:20:49.293129 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:49.293134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:49.294335 (+ 1201us) server_negotiation.cc:407] Connection header received
0925 01:20:49.296237 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:49.296242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:49.296334 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:49.296446 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:49.298016 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.299003 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.300809 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.301262 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.301361 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.305198 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:49.305208 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:49.305214 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:49.305263 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:49.309314 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:49.310125 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:49.310134 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:49.310138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:49.310342 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:49.310884 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:49.310888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:49.310890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:49.311170 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:49.311345 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:49.312050 (+ 705us) server_negotiation.cc:299] Negotiation successful
0925 01:20:49.312324 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:20:49.313813 4922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.293402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:60235)
0925 01:20:49.294164 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:49.294203 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:49.295954 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:49.296754 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:49.296763 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:49.297278 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:20:49.297845 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:49.297862 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.299264 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.299269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:49.300653 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:49.300668 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.301502 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.301512 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:49.301872 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.302613 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:20:49.302646 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:20:49.304920 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:20:49.309468 (+ 4548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:49.309476 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:49.309490 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:49.309965 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:49.310473 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:20:49.310481 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:20:49.310486 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:20:49.310751 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:20:49.311303 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:20:49.311312 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:20:49.311674 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 01:20:49.311953 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 01:20:49.312518 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":299,"threads_started":1}
I20260925 01:20:49.315423 4920 heartbeater.cc:347] Connected to a master server at 127.20.102.190:34573
I20260925 01:20:49.315807 4920 heartbeater.cc:464] Registering TS with master...
I20260925 01:20:49.316413 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.298963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58282 (local address 127.20.102.190:34573)
0925 01:20:49.300308 (+ 1345us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:49.300312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:49.300327 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:20:49.300364 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:49.300367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:49.300424 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:49.300507 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:49.302685 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.303422 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.305301 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.305882 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.305981 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.310071 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:49.310080 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:49.310086 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:49.310126 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:49.313470 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:49.314280 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:49.314285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:49.314287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:49.314402 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:49.314873 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:49.314878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:49.314881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:49.315183 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:49.315392 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:49.315887 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 01:20:49.316076 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1260,"thread_start_us":1133,"threads_started":1}
I20260925 01:20:49.316756 4920 heartbeater.cc:511] Master 127.20.102.190:34573 requested a full tablet report, sending...
I20260925 01:20:49.318346 4472 ts_manager.cc:194] Registered new tserver with Master: 3eed8c8ed57f4bd5b74e428ad371dc1c (127.20.102.131:46813)
I20260925 01:20:49.319212 4472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:60235
I20260925 01:20:49.319432 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:20:49.327157 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.320454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58298 (local address 127.20.102.190:34573)
0925 01:20:49.320718 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:49.320726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:49.321126 (+ 400us) server_negotiation.cc:407] Connection header received
0925 01:20:49.321448 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:49.321454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:49.321606 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:49.321774 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:20:49.322302 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.323197 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.325049 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.325516 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.325672 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.326210 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:20:49.326421 (+ 211us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 01:20:49.326849 (+ 428us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58298: token expired
Metrics: {"server-negotiator.queue_time_us":84}
W20260925 01:20:49.327266 4924 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58298: token expired
W20260925 01:20:49.327769 4394 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.20.102.190:34573)
I20260925 01:20:49.341249 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.328265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58306 (local address 127.20.102.190:34573)
0925 01:20:49.328471 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:49.328477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:49.328649 (+ 172us) server_negotiation.cc:407] Connection header received
0925 01:20:49.328883 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:49.328890 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:49.328959 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:49.329076 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:49.330670 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.331422 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.332705 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.333100 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.333256 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.335893 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:49.335905 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:49.335908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:49.335943 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:49.338510 (+ 2567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:49.339202 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:49.339207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:49.339217 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:49.339368 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:49.339761 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:49.339765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:49.339767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:49.340090 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:49.340355 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:49.340660 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 01:20:49.340915 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 01:20:49.345060 4472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:20:49.354967 4472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:20:49.391198 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.374237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37326 (local address 127.20.102.130:44201)
0925 01:20:49.382440 (+ 8203us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:49.382457 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:49.382482 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:20:49.382626 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:49.382636 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:49.382946 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:49.383169 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:49.383972 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.385619 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.388624 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.389967 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.390165 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.390409 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:49.390674 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 01:20:49.390852 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8040,"thread_start_us":7900,"threads_started":1}
I20260925 01:20:49.392521 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.374102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:44201 (local address 127.0.0.1:37326)
0925 01:20:49.374922 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:49.374972 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:49.375195 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:49.383196 (+ 8001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:49.383203 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:49.383235 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:49.383754 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:49.383769 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.385780 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.385784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:49.388396 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:49.388417 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.390212 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.390222 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:49.390405 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.390462 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 01:20:49.390629 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 01:20:49.390827 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":330,"threads_started":1}
I20260925 01:20:49.394387 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.375294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39765 (local address 127.0.0.1:41460)
0925 01:20:49.381676 (+ 6382us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:49.381705 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:49.381863 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:49.386342 (+ 4479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:49.386346 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:49.386375 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:49.386778 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:49.386790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.388898 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.388903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:49.391550 (+ 2647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:49.391569 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.393714 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.393721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:49.393915 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.393956 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 01:20:49.394021 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 01:20:49.394130 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6200,"mutex_wait_us":65,"thread_start_us":6039,"threads_started":1}
I20260925 01:20:49.395367 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.375892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.20.102.129:39765)
0925 01:20:49.384572 (+ 8680us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:49.384581 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:49.384610 (+ 29us) server_negotiation.cc:407] Connection header received
0925 01:20:49.384730 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:49.384738 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:49.386118 (+ 1380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:49.386976 (+ 858us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:49.387246 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.388752 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.391711 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.393406 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.394797 (+ 1391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.394931 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:49.394989 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 01:20:49.395117 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8512,"thread_start_us":8354,"threads_started":1}
I20260925 01:20:49.397411 4724 tablet_service.cc:1467] Processing CreateTablet for tablet fe152a354d73478ba658a7bde47f17b7 (DEFAULT_TABLE table=test-table [id=af2db16c4db149aaa33159514fdc7e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:20:49.399271 4724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe152a354d73478ba658a7bde47f17b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:20:49.399127 4592 tablet_service.cc:1467] Processing CreateTablet for tablet fe152a354d73478ba658a7bde47f17b7 (DEFAULT_TABLE table=test-table [id=af2db16c4db149aaa33159514fdc7e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:20:49.400554 4592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe152a354d73478ba658a7bde47f17b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:20:49.401633 4657 heartbeater.cc:503] Master 127.20.102.190:34573 was elected leader, sending a full tablet report...
I20260925 01:20:49.402745 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.383057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:46813 (local address 127.0.0.1:58246)
0925 01:20:49.383882 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:49.383901 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:49.384012 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:49.385764 (+ 1752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:49.385772 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:49.385807 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:49.386292 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:49.386307 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.388807 (+ 2500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.388815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:49.396545 (+ 7730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:49.396563 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.401394 (+ 4831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.401401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:49.402103 (+ 702us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.402140 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 01:20:49.402242 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 01:20:49.402409 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":502,"threads_started":1}
I20260925 01:20:49.403609 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.382170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.20.102.131:46813)
0925 01:20:49.383022 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:49.383030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:49.384775 (+ 1745us) server_negotiation.cc:407] Connection header received
0925 01:20:49.384896 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:49.384907 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:49.385159 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:49.385356 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:49.386557 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.388556 (+ 1999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.396797 (+ 8241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.400813 (+ 4016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.401020 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.401235 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:49.403026 (+ 1791us) server_negotiation.cc:299] Negotiation successful
0925 01:20:49.403221 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":491,"threads_started":1}
I20260925 01:20:49.408054 4855 tablet_service.cc:1467] Processing CreateTablet for tablet fe152a354d73478ba658a7bde47f17b7 (DEFAULT_TABLE table=test-table [id=af2db16c4db149aaa33159514fdc7e56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:20:49.409971 4855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe152a354d73478ba658a7bde47f17b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:20:49.425437 4937 tablet_bootstrap.cc:492] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073: Bootstrap starting.
I20260925 01:20:49.426469 4936 tablet_bootstrap.cc:492] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Bootstrap starting.
I20260925 01:20:49.429420 4937 tablet_bootstrap.cc:654] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073: Neither blocks nor log segments found. Creating new log.
I20260925 01:20:49.430146 4936 tablet_bootstrap.cc:654] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Neither blocks nor log segments found. Creating new log.
I20260925 01:20:49.431104 4937 log.cc:824] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:49.431494 4936 log.cc:824] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:49.436379 4938 tablet_bootstrap.cc:492] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Bootstrap starting.
I20260925 01:20:49.442627 4938 tablet_bootstrap.cc:654] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Neither blocks nor log segments found. Creating new log.
I20260925 01:20:49.444015 4938 log.cc:824] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Log is configured to *not* fsync() on all Append() calls
I20260925 01:20:49.444971 4936 tablet_bootstrap.cc:492] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd: No bootstrap required, opened a new log
I20260925 01:20:49.445010 4937 tablet_bootstrap.cc:492] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073: No bootstrap required, opened a new log
I20260925 01:20:49.445365 4936 ts_tablet_manager.cc:1397] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.004s
I20260925 01:20:49.445425 4937 ts_tablet_manager.cc:1397] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.011s
I20260925 01:20:49.458534 4936 raft_consensus.cc:348] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } } peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } }
I20260925 01:20:49.459139 4936 raft_consensus.cc:374] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:20:49.458804 4937 raft_consensus.cc:348] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } } peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } }
I20260925 01:20:49.459290 4936 raft_consensus.cc:729] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f1c3572a5c41d4b81ad9fd41c69ccd, State: Initialized, Role: FOLLOWER
I20260925 01:20:49.459498 4937 raft_consensus.cc:374] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:20:49.459697 4938 tablet_bootstrap.cc:492] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c: No bootstrap required, opened a new log
I20260925 01:20:49.459681 4937 raft_consensus.cc:729] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2517a06cabfe4529868fccb0380b1073, State: Initialized, Role: FOLLOWER
I20260925 01:20:49.460076 4938 ts_tablet_manager.cc:1397] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.017s
I20260925 01:20:49.460060 4936 consensus_queue.cc:260] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } } peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } }
I20260925 01:20:49.460546 4937 consensus_queue.cc:260] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } } peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } }
I20260925 01:20:49.467954 4789 heartbeater.cc:503] Master 127.20.102.190:34573 was elected leader, sending a full tablet report...
I20260925 01:20:49.469092 4937 ts_tablet_manager.cc:1428] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073: Time spent starting tablet: real 0.023s user 0.021s sys 0.001s
I20260925 01:20:49.474431 4936 ts_tablet_manager.cc:1428] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Time spent starting tablet: real 0.029s user 0.023s sys 0.004s
I20260925 01:20:49.475174 4938 raft_consensus.cc:348] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } } peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } }
I20260925 01:20:49.475788 4938 raft_consensus.cc:374] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:20:49.475950 4938 raft_consensus.cc:729] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eed8c8ed57f4bd5b74e428ad371dc1c, State: Initialized, Role: FOLLOWER
I20260925 01:20:49.476819 4938 consensus_queue.cc:260] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } } peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } }
I20260925 01:20:49.478127 4920 heartbeater.cc:503] Master 127.20.102.190:34573 was elected leader, sending a full tablet report...
I20260925 01:20:49.478828 4938 ts_tablet_manager.cc:1428] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Time spent starting tablet: real 0.019s user 0.008s sys 0.007s
W20260925 01:20:49.541855 4921 tablet.cc:2369] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:20:49.573055 4790 tablet.cc:2369] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:20:49.616573 4658 tablet.cc:2369] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:20:49.724941 4943 raft_consensus.cc:482] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:20:49.725323 4943 raft_consensus.cc:504] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } } peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } }
I20260925 01:20:49.727058 4943 leader_election.cc:288] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0f1c3572a5c41d4b81ad9fd41c69ccd (127.20.102.129:39765), 3eed8c8ed57f4bd5b74e428ad371dc1c (127.20.102.131:46813)
I20260925 01:20:49.734050 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.727460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:49031 (local address 127.20.102.129:39765)
0925 01:20:49.727686 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:49.727691 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:49.728157 (+ 466us) server_negotiation.cc:407] Connection header received
0925 01:20:49.728513 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:49.728518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:49.728612 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:49.728669 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:49.729408 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.730193 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.732284 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.733123 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.733307 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.733388 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:49.733667 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 01:20:49.733794 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:20:49.734145 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.727344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:39765 (local address 127.20.102.130:49031)
0925 01:20:49.728031 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:49.728051 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:49.728305 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:49.728940 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:49.728944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:49.728967 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:49.729239 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:49.729247 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.730331 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.730336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:49.732123 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:49.732139 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.733342 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.733349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:49.733505 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.733529 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:20:49.733647 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 01:20:49.733776 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":329,"threads_started":1}
I20260925 01:20:49.734392 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.728392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:40861 (local address 127.20.102.131:46813)
0925 01:20:49.728634 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:49.728640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:49.728885 (+ 245us) server_negotiation.cc:407] Connection header received
0925 01:20:49.729117 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:49.729122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:49.729188 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:49.729285 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:20:49.729909 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.730776 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.732720 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.733479 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.733676 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.733749 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:49.734047 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 01:20:49.734162 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 01:20:49.734760 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.728103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:46813 (local address 127.20.102.130:40861)
0925 01:20:49.728779 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 01:20:49.728794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:20:49.728959 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:20:49.729301 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:20:49.729305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:20:49.729329 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:20:49.729762 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:49.729771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.730906 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.730911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:49.732528 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:20:49.732542 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.733706 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.733713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:20:49.733883 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.733909 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 01:20:49.734009 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 01:20:49.734443 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":298,"threads_started":1}
I20260925 01:20:49.735459 4612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe152a354d73478ba658a7bde47f17b7" candidate_uuid: "2517a06cabfe4529868fccb0380b1073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" is_pre_election: true
I20260925 01:20:49.735900 4875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe152a354d73478ba658a7bde47f17b7" candidate_uuid: "2517a06cabfe4529868fccb0380b1073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" is_pre_election: true
I20260925 01:20:49.736037 4612 raft_consensus.cc:2454] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2517a06cabfe4529868fccb0380b1073 in term 0.
I20260925 01:20:49.736368 4875 raft_consensus.cc:2454] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2517a06cabfe4529868fccb0380b1073 in term 0.
I20260925 01:20:49.736807 4680 leader_election.cc:302] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2517a06cabfe4529868fccb0380b1073, d0f1c3572a5c41d4b81ad9fd41c69ccd; no voters:
I20260925 01:20:49.737231 4943 raft_consensus.cc:2781] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:20:49.737385 4943 raft_consensus.cc:482] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:20:49.737522 4943 raft_consensus.cc:3037] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:20:49.739766 4943 raft_consensus.cc:504] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } } peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } }
I20260925 01:20:49.740331 4943 leader_election.cc:288] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [CANDIDATE]: Term 1 election: Requested vote from peers d0f1c3572a5c41d4b81ad9fd41c69ccd (127.20.102.129:39765), 3eed8c8ed57f4bd5b74e428ad371dc1c (127.20.102.131:46813)
I20260925 01:20:49.740847 4612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe152a354d73478ba658a7bde47f17b7" candidate_uuid: "2517a06cabfe4529868fccb0380b1073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd"
I20260925 01:20:49.740906 4875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe152a354d73478ba658a7bde47f17b7" candidate_uuid: "2517a06cabfe4529868fccb0380b1073" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c"
I20260925 01:20:49.741039 4612 raft_consensus.cc:3037] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:20:49.741117 4875 raft_consensus.cc:3037] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:20:49.743497 4875 raft_consensus.cc:2454] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2517a06cabfe4529868fccb0380b1073 in term 1.
I20260925 01:20:49.743502 4612 raft_consensus.cc:2454] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2517a06cabfe4529868fccb0380b1073 in term 1.
I20260925 01:20:49.743917 4680 leader_election.cc:302] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2517a06cabfe4529868fccb0380b1073, 3eed8c8ed57f4bd5b74e428ad371dc1c; no voters:
I20260925 01:20:49.744262 4943 raft_consensus.cc:2781] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:20:49.744807 4943 raft_consensus.cc:686] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [term 1 LEADER]: Becoming Leader. State: Replica: 2517a06cabfe4529868fccb0380b1073, State: Running, Role: LEADER
I20260925 01:20:49.745370 4943 consensus_queue.cc:237] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } } peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } }
I20260925 01:20:49.750026 4472 catalog_manager.cc:5495] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2517a06cabfe4529868fccb0380b1073 (127.20.102.130). New cstate: current_term: 1 leader_uuid: "2517a06cabfe4529868fccb0380b1073" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2517a06cabfe4529868fccb0380b1073" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 44201 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 } health_report { overall_health: UNKNOWN } } }
I20260925 01:20:49.836930 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.830987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.20.102.130:44201)
0925 01:20:49.831222 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:49.831230 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:49.831251 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:20:49.831327 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:49.831331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:49.831402 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:49.831582 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:20:49.832041 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.832927 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.834828 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.835251 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.835386 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.835881 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:20:49.836341 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:20:49.836511 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:49.836564 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 01:20:49.836643 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:20:49.845229 4612 raft_consensus.cc:1261] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 1 FOLLOWER]: Refusing update from remote peer 2517a06cabfe4529868fccb0380b1073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:20:49.845259 4875 raft_consensus.cc:1261] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 1 FOLLOWER]: Refusing update from remote peer 2517a06cabfe4529868fccb0380b1073: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:20:49.846179 4943 consensus_queue.cc:1035] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:20:49.846511 4947 consensus_queue.cc:1035] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 39765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:20:49.872938 4472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:20:49.873243 4472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:20:49.877075 4472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Sending DeleteTablet for 3 replicas of tablet fe152a354d73478ba658a7bde47f17b7
I20260925 01:20:49.878715 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 4509
I20260925 01:20:49.879153 4592 tablet_service.cc:1514] Processing DeleteTablet for tablet fe152a354d73478ba658a7bde47f17b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:20:49 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:41460
I20260925 01:20:49.879530 4724 tablet_service.cc:1514] Processing DeleteTablet for tablet fe152a354d73478ba658a7bde47f17b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:20:49 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:37326
I20260925 01:20:49.879915 4855 tablet_service.cc:1514] Processing DeleteTablet for tablet fe152a354d73478ba658a7bde47f17b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:20:49 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:58246
I20260925 01:20:49.904057 4959 tablet_replica.cc:330] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073: stopping tablet replica
I20260925 01:20:49.905896 4959 raft_consensus.cc:2229] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:20:49.907764 4959 raft_consensus.cc:2258] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:20:49.910810 4958 tablet_replica.cc:330] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd: stopping tablet replica
I20260925 01:20:49.911352 4958 raft_consensus.cc:2229] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:20:49.912124 4960 tablet_replica.cc:330] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c: stopping tablet replica
I20260925 01:20:49.912632 4960 raft_consensus.cc:2229] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:20:49.913290 4960 raft_consensus.cc:2258] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:20:49.914096 4958 raft_consensus.cc:2258] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:20:49.915438 4959 ts_tablet_manager.cc:1905] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:20:49.916599 4958 ts_tablet_manager.cc:1905] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:20:49.919622 4960 ts_tablet_manager.cc:1905] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:20:49.923985 4959 ts_tablet_manager.cc:1918] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:20:49.924105 4958 ts_tablet_manager.cc:1918] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:20:49.924250 4959 log.cc:1196] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/fe152a354d73478ba658a7bde47f17b7
I20260925 01:20:49.924342 4958 log.cc:1196] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/fe152a354d73478ba658a7bde47f17b7
I20260925 01:20:49.924948 4958 ts_tablet_manager.cc:1939] T fe152a354d73478ba658a7bde47f17b7 P d0f1c3572a5c41d4b81ad9fd41c69ccd: Deleting consensus metadata
I20260925 01:20:49.926288 4959 ts_tablet_manager.cc:1939] T fe152a354d73478ba658a7bde47f17b7 P 2517a06cabfe4529868fccb0380b1073: Deleting consensus metadata
I20260925 01:20:49.926954 4459 catalog_manager.cc:4841] TS d0f1c3572a5c41d4b81ad9fd41c69ccd (127.20.102.129:39765): tablet fe152a354d73478ba658a7bde47f17b7 (table test-table [id=af2db16c4db149aaa33159514fdc7e56]) successfully deleted
I20260925 01:20:49.927983 4960 ts_tablet_manager.cc:1918] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:20:49.928180 4960 log.cc:1196] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/fe152a354d73478ba658a7bde47f17b7
I20260925 01:20:49.928781 4960 ts_tablet_manager.cc:1939] T fe152a354d73478ba658a7bde47f17b7 P 3eed8c8ed57f4bd5b74e428ad371dc1c: Deleting consensus metadata
I20260925 01:20:49.929137 4459 catalog_manager.cc:4841] TS 2517a06cabfe4529868fccb0380b1073 (127.20.102.130:44201): tablet fe152a354d73478ba658a7bde47f17b7 (table test-table [id=af2db16c4db149aaa33159514fdc7e56]) successfully deleted
I20260925 01:20:49.930867 4459 catalog_manager.cc:4841] TS 3eed8c8ed57f4bd5b74e428ad371dc1c (127.20.102.131:46813): tablet fe152a354d73478ba658a7bde47f17b7 (table test-table [id=af2db16c4db149aaa33159514fdc7e56]) successfully deleted
I20260925 01:20:49.933351 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:49.880405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41476 (local address 127.20.102.129:39765)
0925 01:20:49.880609 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:49.880617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:49.902423 (+ 21806us) server_negotiation.cc:407] Connection header received
0925 01:20:49.902703 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:49.902710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:49.902813 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:49.902947 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:49.909784 (+ 6837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.911187 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.913274 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:49.914120 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:49.914273 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:49.925309 (+ 11036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:49.925325 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:49.925344 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:49.925399 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:49.929997 (+ 4598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:49.930890 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:49.930900 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:49.930904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:49.931110 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:49.931496 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:49.931502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:49.931506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:49.932037 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:49.932374 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:49.932658 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 01:20:49.932924 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:20:49.934031 4652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:50.025344 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 4509
I20260925 01:20:50.053138 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 4661
I20260925 01:20:50.071293 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:50.053981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37350 (local address 127.20.102.130:44201)
0925 01:20:50.054186 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:50.054191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:50.054275 (+ 84us) server_negotiation.cc:407] Connection header received
0925 01:20:50.054491 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:50.054495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:50.054610 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:50.054703 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:50.056665 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:50.057656 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:50.059776 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:50.060281 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:50.060387 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:50.064120 (+ 3733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:50.064134 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:50.064151 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:50.064186 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:50.068105 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:50.069129 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:50.069138 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:50.069140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:50.069297 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:50.069733 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:50.069737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:50.069739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:50.070113 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:50.070395 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:50.070769 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 01:20:50.070940 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:20:50.072065 4784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:50.187664 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 4661
I20260925 01:20:50.207494 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 4792
I20260925 01:20:50.220737 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:20:50.208188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.20.102.131:46813)
0925 01:20:50.208354 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 01:20:50.208359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:20:50.208376 (+ 17us) server_negotiation.cc:407] Connection header received
0925 01:20:50.208524 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:20:50.208539 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:20:50.208611 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:20:50.208760 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:20:50.210085 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:50.210807 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:50.212253 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:20:50.212645 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:20:50.212836 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:20:50.215241 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:20:50.215255 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:20:50.215272 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:20:50.215307 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:20:50.218049 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:50.218702 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:50.218710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:50.218712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:50.218839 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:20:50.219197 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:20:50.219200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:20:50.219202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:20:50.219570 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:20:50.219787 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 01:20:50.220265 (+ 478us) server_negotiation.cc:299] Negotiation successful
0925 01:20:50.220424 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:20:50.221369 4915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:50.310776 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 4792
I20260925 01:20:50.328532 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 4439
I20260925 01:20:50.329821 4502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:20:50.445209 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 4439
I20260925 01:21:10.468588 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:34573
--webserver_interface=127.20.102.190
--webserver_port=40903
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:34573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:10.805847 4966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:10.806159 4966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:10.806253 4966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:10.813575 4966 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 01:21:10.813673 4966 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 01:21:10.813711 4966 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 01:21:10.813755 4966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:21:10.813808 4966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:10.813874 4966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:10.813920 4966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:21:10.813966 4966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:21:10.824441 4966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:34573
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:34573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=40903
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:10.825699 4966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:10.826998 4966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:10.838580 4971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:10.838619 4972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:10.839308 4974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:10.839833 4966 server_base.cc:1034] running on GCE node
I20260925 01:21:10.840642 4966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:10.842077 4966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:10.843327 4966 hybrid_clock.cc:648] HybridClock initialized: now 1790299270843260 us; error 43 us; skew 500 ppm
I20260925 01:21:10.843663 4966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:10 dist-test-slave-s12s krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299270, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:10.853207 4966 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:21:10.854749 4966 webserver.cc:460] Webserver started at http://127.20.102.190:40903/ using document root <none> and password file <none>
I20260925 01:21:10.855474 4966 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:10.855616 4966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:10.861141 4966 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 01:21:10.864030 4981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:10.865450 4966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:21:10.865656 4966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "d5b3336265ff45a5bd6d02a5d4362080"
format_stamp: "Formatted at 2026-09-25 01:20:24 on dist-test-slave-s12s"
server_key: "0e2cf53bdc27346be784a4c63deb769a"
server_key_iv: "7711f02f98c41c24d4922eb95e86e3c7"
server_key_version: "encryptionkey@0"
I20260925 01:21:10.866671 4966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:10.878974 4966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:10.883558 4966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:10.883843 4966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:10.902652 4966 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:34573
I20260925 01:21:10.902736 5032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:34573 every 8 connection(s)
I20260925 01:21:10.904131 4966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:21:10.911144 5033 sys_catalog.cc:263] Verifying existing consensus state
I20260925 01:21:10.912940 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 4966
I20260925 01:21:10.913954 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:39765
--local_ip_for_outbound_sockets=127.20.102.129
--tserver_master_addrs=127.20.102.190:34573
--webserver_port=43929
--webserver_interface=127.20.102.129
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:21:10.915515 5033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Bootstrap starting.
I20260925 01:21:10.938282 5033 log.cc:824] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:10.958648 5033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 01:21:10.959231 5033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Bootstrap complete.
I20260925 01:21:10.968652 5033 raft_consensus.cc:348] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } }
I20260925 01:21:10.969646 5033 raft_consensus.cc:729] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b3336265ff45a5bd6d02a5d4362080, State: Initialized, Role: FOLLOWER
I20260925 01:21:10.970305 5033 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } }
I20260925 01:21:10.970539 5033 raft_consensus.cc:388] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:21:10.970687 5033 raft_consensus.cc:482] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:21:10.970885 5033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 2 FOLLOWER]: Advancing to term 3
I20260925 01:21:10.974923 5033 raft_consensus.cc:504] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } }
I20260925 01:21:10.975449 5033 leader_election.cc:302] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5b3336265ff45a5bd6d02a5d4362080; no voters:
I20260925 01:21:10.976202 5033 leader_election.cc:288] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 01:21:10.976346 5037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 01:21:10.977643 5037 raft_consensus.cc:686] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [term 3 LEADER]: Becoming Leader. State: Replica: d5b3336265ff45a5bd6d02a5d4362080, State: Running, Role: LEADER
I20260925 01:21:10.978304 5037 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } }
I20260925 01:21:10.978576 5033 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:21:10.981257 5039 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5b3336265ff45a5bd6d02a5d4362080. Latest consensus state: current_term: 3 leader_uuid: "d5b3336265ff45a5bd6d02a5d4362080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } } }
I20260925 01:21:10.981245 5038 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d5b3336265ff45a5bd6d02a5d4362080" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5b3336265ff45a5bd6d02a5d4362080" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34573 } } }
I20260925 01:21:10.981735 5039 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:10.981740 5038 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:10.994197 5043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:21:11.004066 5043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:21:11.005887 5043 catalog_manager.cc:1259] Loaded cluster ID: d798082d39e846d2b073f16a607e911e
I20260925 01:21:11.006124 5043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:21:11.010440 5043 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:21:11.012228 5043 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Loaded TSK: 0
I20260925 01:21:11.012503 5043 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d5b3336265ff45a5bd6d02a5d4362080: Loaded TSK: 1
F20260925 01:21:11.013972 5043 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fad541b19ad google::LogMessage::Fail() at ??:0
@ 0x7fad541b488a google::LogMessage::SendToLog() at ??:0
@ 0x7fad541b14de google::LogMessage::Flush() at ??:0
@ 0x7fad541b2a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fad57f157d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fad57f0473e kudu::security::TokenSigner::ImportKeys() at ??:0
I20260925 01:21:11.036424 5054 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fad69be01c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fad69bdf783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fad69c6b2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fad69c6b0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fad69cb55e9 std::function<>::operator()() at ??:0
@ 0x7fad69be3cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fad69be1c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fad69c69610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fad69c69439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fad56dfd345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fad56e0091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fad56e00799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fad56ddbeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fad5a0b56db start_thread at ??:0
@ 0x7fad5304b61f clone at ??:0
W20260925 01:21:11.330183 5035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:11.330525 5035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:11.330734 5035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:11.338351 5035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:11.338464 5035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:11.338660 5035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:21:11.348752 5035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:39765
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=43929
--tserver_master_addrs=127.20.102.190:34573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:11.350108 5035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:11.351552 5035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:11.364640 5080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:11.364669 5081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:11.364975 5083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:11.365623 5035 server_base.cc:1034] running on GCE node
I20260925 01:21:11.366273 5035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:11.367296 5035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:11.368563 5035 hybrid_clock.cc:648] HybridClock initialized: now 1790299271368504 us; error 53 us; skew 500 ppm
I20260925 01:21:11.368906 5035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:11 dist-test-slave-s12s krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299271, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:11.372009 5035 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:21:11.373648 5035 webserver.cc:460] Webserver started at http://127.20.102.129:43929/ using document root <none> and password file <none>
I20260925 01:21:11.374327 5035 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:11.374475 5035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:11.379638 5035 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 01:21:11.382378 5090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:11.383713 5035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:21:11.383845 5035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "d0f1c3572a5c41d4b81ad9fd41c69ccd"
format_stamp: "Formatted at 2026-09-25 01:20:25 on dist-test-slave-s12s"
server_key: "a3df6ab232db08252541af711e7bd9de"
server_key_iv: "365a509c3422c33f6ec90af5d2fb94bd"
server_key_version: "encryptionkey@0"
I20260925 01:21:11.384836 5035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:11.400810 5035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:11.405218 5035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:11.405447 5035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:11.406491 5035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:11.407939 5035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:11.408028 5035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:11.408102 5035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:11.408160 5035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:11.442224 5035 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:39765
I20260925 01:21:11.442312 5202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:39765 every 8 connection(s)
I20260925 01:21:11.443735 5035 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:21:11.448570 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:11.446768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:40561)
0925 01:21:11.447487 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:11.447745 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":288,"threads_started":1}
W20260925 01:21:11.449435 5203 heartbeater.cc:644] Failed to heartbeat to 127.20.102.190:34573 (0 consecutive failures): Network error: Failed to ping master at 127.20.102.190:34573: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
I20260925 01:21:11.450210 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:11.449902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:51711)
0925 01:21:11.450047 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:11.450112 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 01:21:11.451129 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:11.450800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:60613)
0925 01:21:11.450944 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:11.451021 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 01:21:11.451433 5203 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 01:21:11.453512 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 5035
I20260925 01:21:11.454157 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:44201
--local_ip_for_outbound_sockets=127.20.102.130
--tserver_master_addrs=127.20.102.190:34573
--webserver_port=41457
--webserver_interface=127.20.102.130
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:11.789701 5206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:11.789989 5206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:11.790086 5206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:11.797299 5206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:11.797408 5206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:11.797526 5206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:21:11.807475 5206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:44201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=41457
--tserver_master_addrs=127.20.102.190:34573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:11.808794 5206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:11.810079 5206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:11.823570 5211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:11.823530 5214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:11.823611 5212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:11.824784 5206 server_base.cc:1034] running on GCE node
I20260925 01:21:11.825371 5206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:11.826320 5206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:11.827518 5206 hybrid_clock.cc:648] HybridClock initialized: now 1790299271827426 us; error 84 us; skew 500 ppm
I20260925 01:21:11.827817 5206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:11 dist-test-slave-s12s krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299271, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:11.831115 5206 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:21:11.832712 5206 webserver.cc:460] Webserver started at http://127.20.102.130:41457/ using document root <none> and password file <none>
I20260925 01:21:11.833412 5206 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:11.833515 5206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:11.838928 5206 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 01:21:11.841686 5221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:11.843035 5206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:21:11.843226 5206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "2517a06cabfe4529868fccb0380b1073"
format_stamp: "Formatted at 2026-09-25 01:20:25 on dist-test-slave-s12s"
server_key: "416c21a236f550e3ab64553776960eac"
server_key_iv: "15fb1b40822197f0c76d69ec36769e68"
server_key_version: "encryptionkey@0"
I20260925 01:21:11.844218 5206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:11.863467 5206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:11.868109 5206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:11.868402 5206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:11.869468 5206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:11.870886 5206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:11.870976 5206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:11.871151 5206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:11.871237 5206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:11.906620 5206 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:44201
I20260925 01:21:11.906710 5333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:44201 every 8 connection(s)
I20260925 01:21:11.908140 5206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:21:11.908478 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 5206
I20260925 01:21:11.909322 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:46813
--local_ip_for_outbound_sockets=127.20.102.131
--tserver_master_addrs=127.20.102.190:34573
--webserver_port=42703
--webserver_interface=127.20.102.131
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:21:11.913762 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:11.911609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:35637)
0925 01:21:11.912462 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:11.912799 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":367,"threads_started":1}
W20260925 01:21:11.914426 5334 heartbeater.cc:644] Failed to heartbeat to 127.20.102.190:34573 (0 consecutive failures): Network error: Failed to ping master at 127.20.102.190:34573: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
I20260925 01:21:11.915432 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:11.915051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:59777)
0925 01:21:11.915243 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:11.915317 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 01:21:11.916363 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:11.916046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:45275)
0925 01:21:11.916185 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:11.916258 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 01:21:11.916721 5334 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 01:21:12.249785 5337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:12.250108 5337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:12.250352 5337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:12.257753 5337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:12.257874 5337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:12.258075 5337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:21:12.268008 5337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:46813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=42703
--tserver_master_addrs=127.20.102.190:34573
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:12.269520 5337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:12.270917 5337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:12.285027 5342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:12.285068 5345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:12.284986 5343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:12.286216 5337 server_base.cc:1034] running on GCE node
I20260925 01:21:12.286959 5337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:12.288116 5337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:12.289335 5337 hybrid_clock.cc:648] HybridClock initialized: now 1790299272289294 us; error 43 us; skew 500 ppm
I20260925 01:21:12.289700 5337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:12 dist-test-slave-s12s krb5kdc[3887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299272, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:12.293139 5337 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:21:12.294682 5337 webserver.cc:460] Webserver started at http://127.20.102.131:42703/ using document root <none> and password file <none>
I20260925 01:21:12.295423 5337 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:12.295529 5337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:12.301319 5337 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 01:21:12.304574 5352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:12.306212 5337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:21:12.306413 5337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "3eed8c8ed57f4bd5b74e428ad371dc1c"
format_stamp: "Formatted at 2026-09-25 01:20:26 on dist-test-slave-s12s"
server_key: "7eb50acc54e1cea8cf8ae567abbe6447"
server_key_iv: "c752b6fe65436b88ef862506b27077a6"
server_key_version: "encryptionkey@0"
I20260925 01:21:12.307482 5337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:12.334872 5337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:12.339147 5337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:12.339401 5337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:12.340405 5337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:12.341905 5337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:12.342002 5337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:12.342082 5337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:12.342136 5337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:12.376974 5337 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:46813
I20260925 01:21:12.377071 5464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:46813 every 8 connection(s)
I20260925 01:21:12.378548 5337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:21:12.383684 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:12.381699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:39905)
0925 01:21:12.382484 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:12.382760 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":314,"threads_started":1}
W20260925 01:21:12.384390 5465 heartbeater.cc:644] Failed to heartbeat to 127.20.102.190:34573 (0 consecutive failures): Network error: Failed to ping master at 127.20.102.190:34573: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
I20260925 01:21:12.385363 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:12.384992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:50751)
0925 01:21:12.385158 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:12.385253 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 01:21:12.386117 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 5337
I20260925 01:21:12.386278 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:12.385909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:34707)
0925 01:21:12.386055 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:12.386137 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 01:21:12.386535 5465 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 01:21:12.452965 5469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:12.452018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:38719)
0925 01:21:12.452542 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:12.452663 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":252,"threads_started":1}
I20260925 01:21:12.918080 5470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:12.917251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:47651)
0925 01:21:12.917746 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:12.917855 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":236,"threads_started":1}
I20260925 01:21:13.388296 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:13.387189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:36497)
0925 01:21:13.387778 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:13.387941 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":255,"threads_started":1}
I20260925 01:21:13.454667 5472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:13.453918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:37275)
0925 01:21:13.454387 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:13.454471 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":234,"threads_started":1}
I20260925 01:21:13.919775 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:13.918995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:45103)
0925 01:21:13.919498 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:13.919595 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":240,"threads_started":1}
I20260925 01:21:14.390043 5474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:14.389318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:42229)
0925 01:21:14.389789 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:14.389872 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":243,"threads_started":1}
I20260925 01:21:14.456368 5475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:14.455627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:49847)
0925 01:21:14.456105 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:14.456193 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":249,"threads_started":1}
I20260925 01:21:14.921622 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:14.920884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:39475)
0925 01:21:14.921347 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:14.921437 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":240,"threads_started":1}
I20260925 01:21:15.391639 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:15.390946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:50367)
0925 01:21:15.391377 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:15.391457 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":223,"threads_started":1}
I20260925 01:21:15.458021 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:15.457311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:39015)
0925 01:21:15.457755 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:15.457835 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":219,"threads_started":1}
I20260925 01:21:15.923316 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:15.922539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:36791)
0925 01:21:15.923021 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:15.923113 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":246,"threads_started":1}
I20260925 01:21:16.393410 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:16.392531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:54101)
0925 01:21:16.393129 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:16.393225 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":311,"threads_started":1}
I20260925 01:21:16.459757 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:16.458991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:54737)
0925 01:21:16.459490 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:16.459583 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":261,"threads_started":1}
I20260925 01:21:16.925048 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:16.924270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:39685)
0925 01:21:16.924797 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:16.924880 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":247,"threads_started":1}
I20260925 01:21:17.395215 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:17.394473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:53683)
0925 01:21:17.394943 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:17.395031 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":243,"threads_started":1}
I20260925 01:21:17.461596 5484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:17.460837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:49411)
0925 01:21:17.461313 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:17.461431 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":240,"threads_started":1}
I20260925 01:21:17.926810 5485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:17.926002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:43927)
0925 01:21:17.926547 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:17.926646 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":303,"threads_started":1}
I20260925 01:21:18.397373 5486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:18.396413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:41307)
0925 01:21:18.397077 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:18.397200 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":373,"threads_started":1}
I20260925 01:21:18.463570 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:18.462734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:58241)
0925 01:21:18.463260 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:18.463348 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":233,"threads_started":1}
I20260925 01:21:18.928601 5488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:18.927815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:36747)
0925 01:21:18.928300 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:18.928384 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":214,"threads_started":1}
I20260925 01:21:19.399296 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:19.398467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:36399)
0925 01:21:19.399008 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:19.399090 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":234,"threads_started":1}
I20260925 01:21:19.465394 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:19.464530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:35903)
0925 01:21:19.465113 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:19.465217 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":279,"threads_started":1}
I20260925 01:21:19.930369 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:19.929510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:48667)
0925 01:21:19.930037 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:19.930159 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":247,"threads_started":1}
I20260925 01:21:20.400957 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:20.400141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:34113)
0925 01:21:20.400644 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:20.400796 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":253,"threads_started":1}
I20260925 01:21:20.467012 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:20.466276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:36269)
0925 01:21:20.466762 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:20.466841 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":234,"threads_started":1}
I20260925 01:21:20.932283 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:20.931449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:48201)
0925 01:21:20.932006 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:20.932107 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":303,"threads_started":1}
I20260925 01:21:21.402684 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:21.401821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:45381)
0925 01:21:21.402387 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:21.402482 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":286,"threads_started":1}
I20260925 01:21:21.468863 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:21.467924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:33101)
0925 01:21:21.468446 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:21.468530 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":319,"threads_started":1}
I20260925 01:21:21.933972 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:21.933276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:54331)
0925 01:21:21.933714 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:21.933807 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":233,"threads_started":1}
I20260925 01:21:22.404397 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:22.403679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:57979)
0925 01:21:22.404150 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:22.404233 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
I20260925 01:21:22.470569 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:22.469711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:50679)
0925 01:21:22.470238 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:22.470344 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":253,"threads_started":1}
I20260925 01:21:22.935606 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:22.934897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:55429)
0925 01:21:22.935329 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:22.935420 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":215,"threads_started":1}
I20260925 01:21:23.406313 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:23.405508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:35893)
0925 01:21:23.406028 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:23.406135 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":266,"threads_started":1}
I20260925 01:21:23.472231 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:23.471508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:48605)
0925 01:21:23.471979 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:23.472062 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":217,"threads_started":1}
I20260925 01:21:23.937364 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:23.936599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:58253)
0925 01:21:23.937119 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:23.937204 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":283,"threads_started":1}
I20260925 01:21:24.408005 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:24.407309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:50827)
0925 01:21:24.407762 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:24.407849 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":222,"threads_started":1}
I20260925 01:21:24.474061 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:24.473211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:36663)
0925 01:21:24.473686 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:24.473780 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":237,"threads_started":1}
I20260925 01:21:24.939126 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:24.938282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:38351)
0925 01:21:24.938806 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:24.938910 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":260,"threads_started":1}
I20260925 01:21:25.409825 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:25.409004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:51435)
0925 01:21:25.409515 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:25.409642 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":244,"threads_started":1}
I20260925 01:21:25.475845 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:25.475055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:41859)
0925 01:21:25.475523 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:25.475649 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":249,"threads_started":1}
I20260925 01:21:25.940881 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:25.940061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:35435)
0925 01:21:25.940538 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:25.940623 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":242,"threads_started":1}
I20260925 01:21:26.411589 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:26.410737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:44607)
0925 01:21:26.411306 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:26.411395 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":276,"threads_started":1}
I20260925 01:21:26.477838 5511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:26.477080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:59569)
0925 01:21:26.477542 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:26.477652 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":232,"threads_started":1}
I20260925 01:21:26.943017 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:26.942157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.130:32957)
0925 01:21:26.942685 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:26.942805 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":241,"threads_started":1}
I20260925 01:21:27.391083 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 5035
I20260925 01:21:27.409979 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:27.392406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58586 (local address 127.20.102.129:39765)
0925 01:21:27.393081 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:27.393112 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:27.393162 (+ 50us) server_negotiation.cc:407] Connection header received
0925 01:21:27.394669 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:27.394679 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:27.395143 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:27.395527 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:27.397166 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:27.398406 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:27.399800 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:27.400528 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:27.400775 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:27.403523 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:27.403531 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:27.403546 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:27.403590 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:27.406471 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:27.407208 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:27.407214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:27.407222 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:27.407392 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:27.407785 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:27.407791 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:27.407794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:27.408338 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:27.408676 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:27.409199 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 01:21:27.409646 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":297,"threads_started":1}
I20260925 01:21:27.411470 5198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:27.416326 5515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:27.412928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.131:33533)
0925 01:21:27.415983 (+ 3055us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:27.416144 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2914,"thread_start_us":2805,"threads_started":1}
I20260925 01:21:27.496908 5516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:27.495625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34573 (local address 127.20.102.129:33783)
0925 01:21:27.496232 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:27.496535 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.190:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":300,"threads_started":1}
I20260925 01:21:27.497069 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 5035
I20260925 01:21:27.516080 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 5206
I20260925 01:21:27.535105 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:27.517212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49794 (local address 127.20.102.130:44201)
0925 01:21:27.517773 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:27.517787 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:27.517824 (+ 37us) server_negotiation.cc:407] Connection header received
0925 01:21:27.519293 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:27.519302 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:27.519739 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:27.520108 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:27.521693 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:27.522946 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:27.524280 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:27.525048 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:27.525253 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:27.527933 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:27.527941 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:27.527955 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:27.527992 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:27.530990 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:27.531697 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:27.531709 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:27.531716 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:27.531885 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:27.532301 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:27.532305 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:27.532307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:27.533085 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:27.533527 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:27.534318 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 01:21:27.534810 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":277,"threads_started":1}
I20260925 01:21:27.536494 5329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:27.620204 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 5206
I20260925 01:21:27.639639 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 5337
I20260925 01:21:27.657162 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:27.640651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.20.102.131:46813)
0925 01:21:27.641341 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:27.641355 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:27.641399 (+ 44us) server_negotiation.cc:407] Connection header received
0925 01:21:27.643143 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:27.643154 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:27.643596 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:27.643908 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:27.645261 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:27.646506 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:27.647824 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:27.648472 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:27.648614 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:27.651363 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:27.651371 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:27.651387 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:27.651422 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:27.654229 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:27.654897 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:27.654903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:27.654909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:27.655069 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:27.655411 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:27.655414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:27.655416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:27.655910 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:27.656167 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:27.656430 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 01:21:27.656856 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":322,"threads_started":1}
I20260925 01:21:27.658526 5460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:27.743716 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 5337
2026-09-25T01:21:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64753 ms)
[----------] 30 tests from SecurityITest (297712 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:21:27 dist-test-slave-s12s krb5kdc[5523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:21:27 dist-test-slave-s12s krb5kdc[5523](info): set up 2 sockets
Sep 25 01:21:27 dist-test-slave-s12s krb5kdc[5523](info): commencing operation
krb5kdc: starting...
W20260925 01:21:28.861053 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:21:29 dist-test-slave-s12s krb5kdc[5523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:21:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:21:29.120782 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:38677
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:38677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:29.452411 5536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:29.452805 5536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:29.452970 5536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:29.460318 5536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:21:29.460433 5536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:29.460523 5536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:29.460626 5536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:21:29.460736 5536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:21:29.470635 5536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:38677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:38677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:29.471926 5536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:29.473209 5536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:29.486639 5541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:29.486634 5544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:29.486593 5542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:29.487452 5536 server_base.cc:1034] running on GCE node
I20260925 01:21:29.488243 5536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:29.489750 5536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:29.490968 5536 hybrid_clock.cc:648] HybridClock initialized: now 1790299289490896 us; error 58 us; skew 500 ppm
I20260925 01:21:29.491295 5536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:29 dist-test-slave-s12s krb5kdc[5523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299289, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:29.494658 5536 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:21:29.496155 5536 webserver.cc:460] Webserver started at http://127.20.102.190:36233/ using document root <none> and password file <none>
I20260925 01:21:29.496920 5536 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:29.497031 5536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:29.497296 5536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:29.500344 5536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "a6b8741794b4463cafc76d1be6bf117a"
format_stamp: "Formatted at 2026-09-25 01:21:29 on dist-test-slave-s12s"
server_key: "06f6adcf26d55753b36a1e149a29a42f"
server_key_iv: "fc696d0f66b3c54cdc372a6e02ed110a"
server_key_version: "encryptionkey@0"
I20260925 01:21:29.501626 5536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "a6b8741794b4463cafc76d1be6bf117a"
format_stamp: "Formatted at 2026-09-25 01:21:29 on dist-test-slave-s12s"
server_key: "06f6adcf26d55753b36a1e149a29a42f"
server_key_iv: "fc696d0f66b3c54cdc372a6e02ed110a"
server_key_version: "encryptionkey@0"
I20260925 01:21:29.507714 5536 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260925 01:21:29.511497 5551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:29.512984 5536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 01:21:29.513190 5536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "a6b8741794b4463cafc76d1be6bf117a"
format_stamp: "Formatted at 2026-09-25 01:21:29 on dist-test-slave-s12s"
server_key: "06f6adcf26d55753b36a1e149a29a42f"
server_key_iv: "fc696d0f66b3c54cdc372a6e02ed110a"
server_key_version: "encryptionkey@0"
I20260925 01:21:29.513408 5536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:29.529162 5536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:29.533601 5536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:29.533910 5536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:29.552994 5536 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:38677
I20260925 01:21:29.553023 5602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:38677 every 8 connection(s)
I20260925 01:21:29.554447 5536 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:21:29.554847 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 5536
I20260925 01:21:29.555086 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:21:29.556027 20890 external_mini_cluster.cc:1428] Setting key 2cdc87e50cff7d799940343eb0038e05
I20260925 01:21:29.559187 5603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 01:21:29 dist-test-slave-s12s krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:21:29.573335 5603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a: Bootstrap starting.
I20260925 01:21:29.577050 5603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:29.578372 5603 log.cc:824] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:29.581480 5606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:29.558173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.20.102.190:38677)
0925 01:21:29.558981 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:29.558994 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:29.559034 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:21:29.560758 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:29.560781 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:29.561151 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:29.561462 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:29.563348 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:29.564892 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:29.566815 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:29.567579 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:29.567727 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:29.573078 (+ 5351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:29.573087 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:29.573105 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:29.573170 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:29.576484 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:29.577470 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:29.577478 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:29.577484 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:29.577689 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:29.578126 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:29.578130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:29.578132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:29.578796 (+ 664us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:29.579194 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:29.579609 (+ 415us) server_negotiation.cc:299] Negotiation successful
0925 01:21:29.580099 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":394,"threads_started":1}
I20260925 01:21:29.581929 5603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a: No bootstrap required, opened a new log
I20260925 01:21:29.590301 5603 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b8741794b4463cafc76d1be6bf117a" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 38677 } }
I20260925 01:21:29.590798 5603 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:29.590941 5603 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6b8741794b4463cafc76d1be6bf117a, State: Initialized, Role: FOLLOWER
I20260925 01:21:29.591536 5603 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b8741794b4463cafc76d1be6bf117a" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 38677 } }
I20260925 01:21:29.591801 5603 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:21:29.591945 5603 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:21:29.592128 5603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:29.594412 5603 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b8741794b4463cafc76d1be6bf117a" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 38677 } }
I20260925 01:21:29.594941 5603 leader_election.cc:302] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6b8741794b4463cafc76d1be6bf117a; no voters:
I20260925 01:21:29.595582 5603 leader_election.cc:288] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:21:29.595755 5608 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:29.596510 5608 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [term 1 LEADER]: Becoming Leader. State: Replica: a6b8741794b4463cafc76d1be6bf117a, State: Running, Role: LEADER
I20260925 01:21:29.597223 5608 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b8741794b4463cafc76d1be6bf117a" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 38677 } }
I20260925 01:21:29.597792 5603 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:21:29.600193 5609 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6b8741794b4463cafc76d1be6bf117a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b8741794b4463cafc76d1be6bf117a" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 38677 } } }
I20260925 01:21:29.600216 5610 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6b8741794b4463cafc76d1be6bf117a. Latest consensus state: current_term: 1 leader_uuid: "a6b8741794b4463cafc76d1be6bf117a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6b8741794b4463cafc76d1be6bf117a" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 38677 } } }
I20260925 01:21:29.600660 5610 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:29.600660 5609 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:29.602864 5614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:21:29.608505 5614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:21:29.618358 5614 catalog_manager.cc:1347] Generated new cluster ID: 9d80f410bfaa4339ab9254a8bfd5c954
I20260925 01:21:29.618482 5614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:21:29.629515 5614 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:21:29.630882 5614 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:21:29.641615 5614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a: Generated new TSK 0
I20260925 01:21:29.642881 5614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:21:29.720858 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:38677
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:30.082007 5629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:30.082278 5629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:30.082373 5629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:30.089663 5629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:30.089773 5629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:30.089903 5629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:21:30.099916 5629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:38677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:30.101263 5629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:30.102598 5629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:30.116205 5634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:30.116117 5635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:30.116245 5637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:30.117154 5629 server_base.cc:1034] running on GCE node
I20260925 01:21:30.117959 5629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:30.119127 5629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:30.120401 5629 hybrid_clock.cc:648] HybridClock initialized: now 1790299290120332 us; error 56 us; skew 500 ppm
I20260925 01:21:30.120867 5629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:30 dist-test-slave-s12s krb5kdc[5523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299290, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:30.124378 5629 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:21:30.126125 5629 webserver.cc:460] Webserver started at http://127.20.102.129:43521/ using document root <none> and password file <none>
I20260925 01:21:30.126864 5629 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:30.126976 5629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:30.127391 5629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:30.130637 5629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd"
format_stamp: "Formatted at 2026-09-25 01:21:30 on dist-test-slave-s12s"
server_key: "6bdc24f41202e09bf93322247ff04dc7"
server_key_iv: "e13397c01d19aa15b60bb0e22936e282"
server_key_version: "encryptionkey@0"
I20260925 01:21:30.131420 5629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd"
format_stamp: "Formatted at 2026-09-25 01:21:30 on dist-test-slave-s12s"
server_key: "6bdc24f41202e09bf93322247ff04dc7"
server_key_iv: "e13397c01d19aa15b60bb0e22936e282"
server_key_version: "encryptionkey@0"
I20260925 01:21:30.137163 5629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 01:21:30.140990 5644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:30.142627 5629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:21:30.142846 5629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd"
format_stamp: "Formatted at 2026-09-25 01:21:30 on dist-test-slave-s12s"
server_key: "6bdc24f41202e09bf93322247ff04dc7"
server_key_iv: "e13397c01d19aa15b60bb0e22936e282"
server_key_version: "encryptionkey@0"
I20260925 01:21:30.143101 5629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:30.159981 5629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:30.164503 5629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:30.164865 5629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:30.165939 5629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:30.167411 5629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:30.167510 5629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:30.167701 5629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:30.167770 5629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:30.204136 5629 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:44577
I20260925 01:21:30.204178 5756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:44577 every 8 connection(s)
I20260925 01:21:30.205652 5629 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:21:30.211205 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 5629
I20260925 01:21:30.211424 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:21:30.212057 20890 external_mini_cluster.cc:1428] Setting key 41f60ede3828cab1d319080e55da67ed
Sep 25 01:21:30 dist-test-slave-s12s krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299290, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:21:30.228093 5759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:30.208295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:36253 (local address 127.20.102.190:38677)
0925 01:21:30.208842 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:30.208847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:30.209801 (+ 954us) server_negotiation.cc:407] Connection header received
0925 01:21:30.211922 (+ 2121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:30.211928 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:30.212039 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:30.212184 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:30.214180 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:30.214958 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:30.216725 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:30.217191 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:30.217345 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:30.221297 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:30.221304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:30.221309 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:30.221356 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:30.224724 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:30.225511 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:30.225517 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:30.225521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:30.225669 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:30.226164 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:30.226170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:30.226174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:30.226468 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:30.226652 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:30.227431 (+ 779us) server_negotiation.cc:299] Negotiation successful
0925 01:21:30.227694 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":281,"threads_started":1}
I20260925 01:21:30.229240 5760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:30.208795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:38677 (local address 127.20.102.129:36253)
0925 01:21:30.209620 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:30.209671 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:30.211633 (+ 1962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:30.212445 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:30.212453 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:30.213144 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:30.213991 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:30.214014 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:30.215146 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:30.215150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:30.216524 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:30.216538 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:30.217459 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:30.217469 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:30.217755 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:30.218595 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:21:30.218621 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:30.221057 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:30.224865 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:30.224875 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:30.224894 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:30.225368 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:30.225795 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:30.225801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:30.225805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:30.226023 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:30.226621 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:30.226630 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:30.227011 (+ 381us) client_negotiation.cc:769] Sending connection context
0925 01:21:30.227316 (+ 305us) client_negotiation.cc:240] Negotiation successful
0925 01:21:30.227892 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":340,"threads_started":1}
I20260925 01:21:30.230938 5757 heartbeater.cc:347] Connected to a master server at 127.20.102.190:38677
I20260925 01:21:30.231279 5757 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:30.232240 5757 heartbeater.cc:511] Master 127.20.102.190:38677 requested a full tablet report, sending...
I20260925 01:21:30.234741 5568 ts_manager.cc:194] Registered new tserver with Master: b2c8c39f4e2c4b16a6590f98f2d0d7cd (127.20.102.129:44577)
I20260925 01:21:30.236400 5568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:36253
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:21:30.268785 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:38677
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:30.610349 5763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:30.610656 5763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:30.610759 5763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:30.619170 5763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:30.619300 5763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:30.619426 5763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:21:30.630300 5763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:38677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:30.631583 5763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:30.633005 5763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:30.647197 5771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:30.647279 5769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:30.647233 5768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:30.650221 5763 server_base.cc:1034] running on GCE node
I20260925 01:21:30.650938 5763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:30.651912 5763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:30.653162 5763 hybrid_clock.cc:648] HybridClock initialized: now 1790299290653108 us; error 45 us; skew 500 ppm
I20260925 01:21:30.653486 5763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:30 dist-test-slave-s12s krb5kdc[5523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299290, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:30.656724 5763 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:21:30.658325 5763 webserver.cc:460] Webserver started at http://127.20.102.130:36039/ using document root <none> and password file <none>
I20260925 01:21:30.659083 5763 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:30.659225 5763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:30.659538 5763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:30.662525 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "9141bdb017f14ed2925f91ad4b51acd2"
format_stamp: "Formatted at 2026-09-25 01:21:30 on dist-test-slave-s12s"
server_key: "5727557994d4b667516af62e4cd0a3a2"
server_key_iv: "1f2534310beb575fe60c5c462a627abc"
server_key_version: "encryptionkey@0"
I20260925 01:21:30.663281 5763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "9141bdb017f14ed2925f91ad4b51acd2"
format_stamp: "Formatted at 2026-09-25 01:21:30 on dist-test-slave-s12s"
server_key: "5727557994d4b667516af62e4cd0a3a2"
server_key_iv: "1f2534310beb575fe60c5c462a627abc"
server_key_version: "encryptionkey@0"
I20260925 01:21:30.668474 5763 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 01:21:30.672005 5778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:30.673514 5763 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:21:30.673692 5763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "9141bdb017f14ed2925f91ad4b51acd2"
format_stamp: "Formatted at 2026-09-25 01:21:30 on dist-test-slave-s12s"
server_key: "5727557994d4b667516af62e4cd0a3a2"
server_key_iv: "1f2534310beb575fe60c5c462a627abc"
server_key_version: "encryptionkey@0"
I20260925 01:21:30.673918 5763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:30.686832 5763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:30.691066 5763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:30.691372 5763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:30.692365 5763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:30.693782 5763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:30.693883 5763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:30.694034 5763 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:30.694116 5763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:30.729450 5763 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:40607
I20260925 01:21:30.729497 5890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:40607 every 8 connection(s)
I20260925 01:21:30.730829 5763 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:21:30.733819 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 5763
I20260925 01:21:30.734012 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:21:30.734558 20890 external_mini_cluster.cc:1428] Setting key 7d0d7f53befe9c4d7b40dc0466fa8988
Sep 25 01:21:30 dist-test-slave-s12s krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299290, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:21:30.752262 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:30.733573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:43295 (local address 127.20.102.190:38677)
0925 01:21:30.734053 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:30.734058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:30.735509 (+ 1451us) server_negotiation.cc:407] Connection header received
0925 01:21:30.737648 (+ 2139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:30.737653 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:30.737764 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:30.737881 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:30.739491 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:30.740236 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:30.741907 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:30.742356 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:30.742499 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:30.746049 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:30.746055 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:30.746058 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:30.746099 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:30.749109 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:30.749869 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:30.749873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:30.749876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:30.750003 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:30.750456 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:30.750465 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:30.750467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:30.750740 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:30.750901 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:30.751622 (+ 721us) server_negotiation.cc:299] Negotiation successful
0925 01:21:30.751869 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":259,"threads_started":1}
I20260925 01:21:30.753358 5894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:30.734240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:38677 (local address 127.20.102.130:43295)
0925 01:21:30.735052 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:30.735098 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:30.737382 (+ 2284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:30.738168 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:30.738177 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:30.738714 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:30.739315 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:30.739333 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:30.740427 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:30.740431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:30.741754 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:30.741767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:30.742595 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:30.742603 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:30.742856 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:30.743594 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:21:30.743618 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:30.745808 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:30.749252 (+ 3444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:30.749262 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:30.749279 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:30.749717 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:30.750118 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:30.750123 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:30.750127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:30.750328 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:30.750876 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:30.750885 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:30.751247 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 01:21:30.751516 (+ 269us) client_negotiation.cc:240] Negotiation successful
0925 01:21:30.752097 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":348,"threads_started":1}
I20260925 01:21:30.754987 5891 heartbeater.cc:347] Connected to a master server at 127.20.102.190:38677
I20260925 01:21:30.755330 5891 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:30.756110 5891 heartbeater.cc:511] Master 127.20.102.190:38677 requested a full tablet report, sending...
I20260925 01:21:30.757829 5568 ts_manager.cc:194] Registered new tserver with Master: 9141bdb017f14ed2925f91ad4b51acd2 (127.20.102.130:40607)
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
I20260925 01:21:30.759117 5568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:43295
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:21:30.792614 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:38677
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:37759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:31.129010 5897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:31.129295 5897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:31.129392 5897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:31.136845 5897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:31.137018 5897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:31.137243 5897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:21:31.147394 5897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:37759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:38677
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:31.148968 5897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:31.150337 5897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:31.163894 5902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:31.163966 5905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:31.163944 5903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:31.164827 5897 server_base.cc:1034] running on GCE node
I20260925 01:21:31.165512 5897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:31.166524 5897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:31.167761 5897 hybrid_clock.cc:648] HybridClock initialized: now 1790299291167700 us; error 56 us; skew 500 ppm
I20260925 01:21:31.168087 5897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:31 dist-test-slave-s12s krb5kdc[5523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299291, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:31.171517 5897 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:21:31.173079 5897 webserver.cc:460] Webserver started at http://127.20.102.131:45519/ using document root <none> and password file <none>
I20260925 01:21:31.173794 5897 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:31.173947 5897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:31.174273 5897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:31.177488 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "f2505639f3e84a8da5e384a306aa62f0"
format_stamp: "Formatted at 2026-09-25 01:21:31 on dist-test-slave-s12s"
server_key: "b48ae573f68cead472ba155d3546f214"
server_key_iv: "69fd92eaa7b8d77b63e61f3106a4c0c7"
server_key_version: "encryptionkey@0"
I20260925 01:21:31.178326 5897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "f2505639f3e84a8da5e384a306aa62f0"
format_stamp: "Formatted at 2026-09-25 01:21:31 on dist-test-slave-s12s"
server_key: "b48ae573f68cead472ba155d3546f214"
server_key_iv: "69fd92eaa7b8d77b63e61f3106a4c0c7"
server_key_version: "encryptionkey@0"
I20260925 01:21:31.183558 5897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:21:31.187340 5912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:31.188818 5897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 01:21:31.188999 5897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "f2505639f3e84a8da5e384a306aa62f0"
format_stamp: "Formatted at 2026-09-25 01:21:31 on dist-test-slave-s12s"
server_key: "b48ae573f68cead472ba155d3546f214"
server_key_iv: "69fd92eaa7b8d77b63e61f3106a4c0c7"
server_key_version: "encryptionkey@0"
I20260925 01:21:31.189206 5897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:31.209390 5897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:31.219295 5897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:31.219614 5897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:31.220716 5897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:31.222180 5897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:31.222283 5897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:31.222437 5897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:31.222512 5897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:31.239799 5757 heartbeater.cc:503] Master 127.20.102.190:38677 was elected leader, sending a full tablet report...
I20260925 01:21:31.258024 5897 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:45475
I20260925 01:21:31.258061 6024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:45475 every 8 connection(s)
I20260925 01:21:31.259493 5897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:21:31.268404 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 5897
I20260925 01:21:31.268626 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:21:31.269328 20890 external_mini_cluster.cc:1428] Setting key 9ea0cf59dca6c0fe58903f771f6cd83e
Sep 25 01:21:31 dist-test-slave-s12s krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299291, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:21:31.281531 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.262536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:55123 (local address 127.20.102.190:38677)
0925 01:21:31.263025 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:31.263029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:31.264092 (+ 1063us) server_negotiation.cc:407] Connection header received
0925 01:21:31.266028 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:31.266033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:31.266106 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:31.266211 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:31.267765 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.268668 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.270900 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.271399 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.271546 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.275297 (+ 3751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:31.275302 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:31.275305 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:31.275341 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:31.278390 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:31.279209 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:31.279214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:31.279216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:31.279355 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:31.279796 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:31.279800 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:31.279803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:31.280111 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:31.280256 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:31.281048 (+ 792us) server_negotiation.cc:299] Negotiation successful
0925 01:21:31.281239 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":231,"threads_started":1}
I20260925 01:21:31.282536 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.263135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:38677 (local address 127.20.102.131:55123)
0925 01:21:31.263907 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:31.263954 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:31.265728 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:31.266431 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:31.266440 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:31.266985 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:31.267577 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:31.267595 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.268883 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.268890 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:31.270704 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:31.270740 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.271711 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.271723 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:31.271949 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.272725 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:21:31.272771 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:31.275054 (+ 2283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:31.278556 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:31.278566 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:31.278587 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:31.279051 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:31.279462 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:31.279465 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:31.279467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:31.279670 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:31.280234 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:31.280241 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:31.280580 (+ 339us) client_negotiation.cc:769] Sending connection context
0925 01:21:31.280937 (+ 357us) client_negotiation.cc:240] Negotiation successful
0925 01:21:31.281354 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":299,"threads_started":1}
I20260925 01:21:31.284359 6025 heartbeater.cc:347] Connected to a master server at 127.20.102.190:38677
I20260925 01:21:31.284771 6025 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:31.285665 6025 heartbeater.cc:511] Master 127.20.102.190:38677 requested a full tablet report, sending...
I20260925 01:21:31.287215 5568 ts_manager.cc:194] Registered new tserver with Master: f2505639f3e84a8da5e384a306aa62f0 (127.20.102.131:45475)
I20260925 01:21:31.288116 5568 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:55123
I20260925 01:21:31.297895 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:21:31.315907 6027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.302648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49136 (local address 127.20.102.190:38677)
0925 01:21:31.302902 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:31.302912 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:31.303330 (+ 418us) server_negotiation.cc:407] Connection header received
0925 01:21:31.303550 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:31.303555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:31.303662 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:31.303774 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:31.305309 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.306056 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.307443 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.307845 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.307949 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.310498 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:31.310503 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:31.310506 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:31.310539 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:31.313384 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:31.314091 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:31.314095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:31.314097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:31.314218 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:31.314535 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:31.314538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:31.314540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:31.314862 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:31.315047 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:31.315360 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 01:21:31.315530 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:21:31.320339 5568 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:21:31.327443 5568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:21:31.356827 6038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.340016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:45475 (local address 127.0.0.1:52022)
0925 01:21:31.340807 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:31.340849 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:31.341011 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:31.342640 (+ 1629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:31.342647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:31.343896 (+ 1249us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:31.344432 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:31.344450 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.348648 (+ 4198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.348657 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:31.353410 (+ 4753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:31.353433 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.355666 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.355672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:31.355890 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.355953 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 01:21:31.356089 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 01:21:31.356310 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":363,"threads_started":1}
I20260925 01:21:31.356972 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.340097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.20.102.131:45475)
0925 01:21:31.341674 (+ 1577us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:31.341685 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:31.341723 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:21:31.341831 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:31.341840 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:31.342107 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:31.342313 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:31.344720 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.348377 (+ 3657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.353699 (+ 5322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.355453 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.355662 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.355849 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:31.356417 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 01:21:31.356602 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1392,"thread_start_us":1246,"threads_started":1}
I20260925 01:21:31.361493 5960 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf4ff83becf4a0ca5871435d587083a (DEFAULT_TABLE table=test-table [id=a69eec96ace34961a6bdf5b4e52259b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:31.362531 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.341209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:40607 (local address 127.0.0.1:44258)
0925 01:21:31.341942 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:31.341965 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:31.342076 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:31.356203 (+ 14127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:31.356209 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:31.356239 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:31.356659 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:31.356671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.358918 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.358926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:31.360238 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:31.360249 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.361622 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.361628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:31.361797 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.361837 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 01:21:31.361945 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 01:21:31.362133 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":470,"threads_started":1}
I20260925 01:21:31.362542 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.341556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44258 (local address 127.20.102.130:40607)
0925 01:21:31.355524 (+ 13968us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:31.355534 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:31.355565 (+ 31us) server_negotiation.cc:407] Connection header received
0925 01:21:31.355672 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:31.355680 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:31.355941 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:31.357018 (+ 1077us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:31.357248 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.358791 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.360398 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.361407 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.361594 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.361785 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:31.362027 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 01:21:31.362211 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13810,"thread_start_us":13669,"threads_started":1}
I20260925 01:21:31.363247 5960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf4ff83becf4a0ca5871435d587083a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:31.366571 6042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.342066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:44577 (local address 127.0.0.1:35572)
0925 01:21:31.354855 (+ 12789us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:31.354888 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:31.355063 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:31.357219 (+ 2156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:31.357224 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:31.358154 (+ 930us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:31.358520 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:31.358531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.361862 (+ 3331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.361867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:31.363898 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:31.363912 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.365792 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.365799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:31.365964 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.366000 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 01:21:31.366092 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 01:21:31.366238 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12606,"thread_start_us":13088,"threads_started":1}
I20260925 01:21:31.366704 6043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.342608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35572 (local address 127.20.102.129:44577)
0925 01:21:31.354869 (+ 12261us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:31.354878 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:31.355410 (+ 532us) server_negotiation.cc:407] Connection header received
0925 01:21:31.355534 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:31.355545 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:31.355854 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:31.356084 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:31.358699 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.360231 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.364050 (+ 3819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.365555 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.365780 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.365960 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:31.366180 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 01:21:31.366359 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12047,"thread_start_us":12289,"threads_started":1}
I20260925 01:21:31.367079 5826 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf4ff83becf4a0ca5871435d587083a (DEFAULT_TABLE table=test-table [id=a69eec96ace34961a6bdf5b4e52259b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:31.368485 5826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf4ff83becf4a0ca5871435d587083a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:31.370939 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf4ff83becf4a0ca5871435d587083a (DEFAULT_TABLE table=test-table [id=a69eec96ace34961a6bdf5b4e52259b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:31.372413 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf4ff83becf4a0ca5871435d587083a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:31.376212 6044 tablet_bootstrap.cc:492] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0: Bootstrap starting.
I20260925 01:21:31.379613 6044 tablet_bootstrap.cc:654] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:31.380662 6044 log.cc:824] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:31.384184 6046 tablet_bootstrap.cc:492] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2: Bootstrap starting.
I20260925 01:21:31.385206 6044 tablet_bootstrap.cc:492] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0: No bootstrap required, opened a new log
I20260925 01:21:31.385493 6044 ts_tablet_manager.cc:1397] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260925 01:21:31.387367 6047 tablet_bootstrap.cc:492] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd: Bootstrap starting.
I20260925 01:21:31.387962 6046 tablet_bootstrap.cc:654] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:31.389070 6046 log.cc:824] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:31.391089 6047 tablet_bootstrap.cc:654] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:31.391741 6046 tablet_bootstrap.cc:492] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2: No bootstrap required, opened a new log
I20260925 01:21:31.392046 6046 ts_tablet_manager.cc:1397] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 01:21:31.392464 6047 log.cc:824] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:31.397684 6044 raft_consensus.cc:348] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2505639f3e84a8da5e384a306aa62f0" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 45475 } } peers { permanent_uuid: "9141bdb017f14ed2925f91ad4b51acd2" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40607 } } peers { permanent_uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44577 } }
I20260925 01:21:31.398182 6044 raft_consensus.cc:374] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:31.398324 6044 raft_consensus.cc:729] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2505639f3e84a8da5e384a306aa62f0, State: Initialized, Role: FOLLOWER
I20260925 01:21:31.399011 6044 consensus_queue.cc:260] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2505639f3e84a8da5e384a306aa62f0" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 45475 } } peers { permanent_uuid: "9141bdb017f14ed2925f91ad4b51acd2" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40607 } } peers { permanent_uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44577 } }
I20260925 01:21:31.400024 6047 tablet_bootstrap.cc:492] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd: No bootstrap required, opened a new log
I20260925 01:21:31.400382 6047 ts_tablet_manager.cc:1397] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20260925 01:21:31.400511 6025 heartbeater.cc:503] Master 127.20.102.190:38677 was elected leader, sending a full tablet report...
I20260925 01:21:31.401306 6044 ts_tablet_manager.cc:1428] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260925 01:21:31.403492 6046 raft_consensus.cc:348] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2505639f3e84a8da5e384a306aa62f0" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 45475 } } peers { permanent_uuid: "9141bdb017f14ed2925f91ad4b51acd2" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40607 } } peers { permanent_uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44577 } }
I20260925 01:21:31.403963 6046 raft_consensus.cc:374] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:31.404116 6046 raft_consensus.cc:729] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9141bdb017f14ed2925f91ad4b51acd2, State: Initialized, Role: FOLLOWER
I20260925 01:21:31.404906 6046 consensus_queue.cc:260] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2505639f3e84a8da5e384a306aa62f0" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 45475 } } peers { permanent_uuid: "9141bdb017f14ed2925f91ad4b51acd2" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40607 } } peers { permanent_uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44577 } }
I20260925 01:21:31.406361 5891 heartbeater.cc:503] Master 127.20.102.190:38677 was elected leader, sending a full tablet report...
I20260925 01:21:31.407364 6046 ts_tablet_manager.cc:1428] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260925 01:21:31.410851 6047 raft_consensus.cc:348] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2505639f3e84a8da5e384a306aa62f0" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 45475 } } peers { permanent_uuid: "9141bdb017f14ed2925f91ad4b51acd2" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40607 } } peers { permanent_uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44577 } }
I20260925 01:21:31.411314 6047 raft_consensus.cc:374] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:31.411424 6047 raft_consensus.cc:729] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2c8c39f4e2c4b16a6590f98f2d0d7cd, State: Initialized, Role: FOLLOWER
I20260925 01:21:31.412086 6047 consensus_queue.cc:260] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2505639f3e84a8da5e384a306aa62f0" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 45475 } } peers { permanent_uuid: "9141bdb017f14ed2925f91ad4b51acd2" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40607 } } peers { permanent_uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44577 } }
I20260925 01:21:31.413775 6047 ts_tablet_manager.cc:1428] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd: Time spent starting tablet: real 0.013s user 0.006s sys 0.006s
W20260925 01:21:31.458525 5758 tablet.cc:2369] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:21:31.482944 5892 tablet.cc:2369] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:21:31.511418 6026 tablet.cc:2369] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:21:31.609987 6050 raft_consensus.cc:482] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:21:31.610430 6050 raft_consensus.cc:504] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2505639f3e84a8da5e384a306aa62f0" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 45475 } } peers { permanent_uuid: "9141bdb017f14ed2925f91ad4b51acd2" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40607 } } peers { permanent_uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44577 } }
I20260925 01:21:31.612201 6050 leader_election.cc:288] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9141bdb017f14ed2925f91ad4b51acd2 (127.20.102.130:40607), b2c8c39f4e2c4b16a6590f98f2d0d7cd (127.20.102.129:44577)
I20260925 01:21:31.618081 6028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.612629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:40607 (local address 127.20.102.131:49377)
0925 01:21:31.612848 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:31.612872 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:31.613005 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:31.613522 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:31.613530 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:31.613581 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:31.613742 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:31.613751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.614796 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.614803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:31.616072 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:31.616084 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.617305 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.617311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:31.617490 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.617537 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 01:21:31.617667 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 01:21:31.617803 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 01:21:31.618093 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.612810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:49377 (local address 127.20.102.130:40607)
0925 01:21:31.613023 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:31.613030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:31.613122 (+ 92us) server_negotiation.cc:407] Connection header received
0925 01:21:31.613213 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:31.613220 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:31.613305 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:31.613399 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:31.613898 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.614641 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.616234 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.617157 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.617319 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.617393 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:31.617692 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 01:21:31.617813 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:21:31.619136 6043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.613366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:33691 (local address 127.20.102.129:44577)
0925 01:21:31.613590 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:31.613596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:31.613937 (+ 341us) server_negotiation.cc:407] Connection header received
0925 01:21:31.614126 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:31.614133 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:31.614196 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:31.614293 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:31.614810 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.615523 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.617297 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.618200 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.618376 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.618458 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:31.618768 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 01:21:31.618896 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 01:21:31.619199 6053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.613074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:44577 (local address 127.20.102.131:33691)
0925 01:21:31.613838 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:31.613852 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:31.613996 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:31.614310 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:31.614314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:31.614340 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:31.614660 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:31.614669 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.615679 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.615684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:31.617152 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:31.617163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.618472 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.618481 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:31.618612 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.618635 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 01:21:31.618741 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 01:21:31.618864 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":531,"threads_started":1}
I20260925 01:21:31.619233 5846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf4ff83becf4a0ca5871435d587083a" candidate_uuid: "f2505639f3e84a8da5e384a306aa62f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9141bdb017f14ed2925f91ad4b51acd2" is_pre_election: true
I20260925 01:21:31.619894 5846 raft_consensus.cc:2454] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2505639f3e84a8da5e384a306aa62f0 in term 0.
I20260925 01:21:31.620319 5712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf4ff83becf4a0ca5871435d587083a" candidate_uuid: "f2505639f3e84a8da5e384a306aa62f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd" is_pre_election: true
I20260925 01:21:31.620651 5916 leader_election.cc:302] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9141bdb017f14ed2925f91ad4b51acd2, f2505639f3e84a8da5e384a306aa62f0; no voters:
I20260925 01:21:31.620918 5712 raft_consensus.cc:2454] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2505639f3e84a8da5e384a306aa62f0 in term 0.
I20260925 01:21:31.621119 6050 raft_consensus.cc:2781] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:21:31.621277 6050 raft_consensus.cc:482] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:21:31.621405 6050 raft_consensus.cc:3037] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:31.623620 6050 raft_consensus.cc:504] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2505639f3e84a8da5e384a306aa62f0" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 45475 } } peers { permanent_uuid: "9141bdb017f14ed2925f91ad4b51acd2" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40607 } } peers { permanent_uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44577 } }
I20260925 01:21:31.624231 6050 leader_election.cc:288] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [CANDIDATE]: Term 1 election: Requested vote from peers 9141bdb017f14ed2925f91ad4b51acd2 (127.20.102.130:40607), b2c8c39f4e2c4b16a6590f98f2d0d7cd (127.20.102.129:44577)
I20260925 01:21:31.624630 5846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf4ff83becf4a0ca5871435d587083a" candidate_uuid: "f2505639f3e84a8da5e384a306aa62f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9141bdb017f14ed2925f91ad4b51acd2"
I20260925 01:21:31.624773 5712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf4ff83becf4a0ca5871435d587083a" candidate_uuid: "f2505639f3e84a8da5e384a306aa62f0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd"
I20260925 01:21:31.624881 5846 raft_consensus.cc:3037] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:31.625010 5712 raft_consensus.cc:3037] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:31.627122 5846 raft_consensus.cc:2454] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2505639f3e84a8da5e384a306aa62f0 in term 1.
I20260925 01:21:31.627364 5712 raft_consensus.cc:2454] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2505639f3e84a8da5e384a306aa62f0 in term 1.
I20260925 01:21:31.627677 5916 leader_election.cc:302] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9141bdb017f14ed2925f91ad4b51acd2, f2505639f3e84a8da5e384a306aa62f0; no voters:
I20260925 01:21:31.628006 6050 raft_consensus.cc:2781] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:31.628481 6050 raft_consensus.cc:686] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [term 1 LEADER]: Becoming Leader. State: Replica: f2505639f3e84a8da5e384a306aa62f0, State: Running, Role: LEADER
I20260925 01:21:31.629045 6050 consensus_queue.cc:237] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2505639f3e84a8da5e384a306aa62f0" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 45475 } } peers { permanent_uuid: "9141bdb017f14ed2925f91ad4b51acd2" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40607 } } peers { permanent_uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44577 } }
I20260925 01:21:31.633492 5566 catalog_manager.cc:5495] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2505639f3e84a8da5e384a306aa62f0 (127.20.102.131). New cstate: current_term: 1 leader_uuid: "f2505639f3e84a8da5e384a306aa62f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2505639f3e84a8da5e384a306aa62f0" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 45475 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9141bdb017f14ed2925f91ad4b51acd2" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44577 } health_report { overall_health: UNKNOWN } } }
I20260925 01:21:31.652187 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.646386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.20.102.131:45475)
0925 01:21:31.646625 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:31.646631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:31.646834 (+ 203us) server_negotiation.cc:407] Connection header received
0925 01:21:31.647090 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:31.647102 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:31.647180 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:31.647336 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:21:31.647724 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.648488 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.650061 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.650454 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.650588 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.651113 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:21:31.651606 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:21:31.651771 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:31.651823 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 01:21:31.651918 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:21:31.660221 5712 raft_consensus.cc:1261] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd [term 1 FOLLOWER]: Refusing update from remote peer f2505639f3e84a8da5e384a306aa62f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:21:31.660585 5846 raft_consensus.cc:1261] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2 [term 1 FOLLOWER]: Refusing update from remote peer f2505639f3e84a8da5e384a306aa62f0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:21:31.661223 6054 consensus_queue.cc:1035] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2c8c39f4e2c4b16a6590f98f2d0d7cd" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:21:31.661648 6050 consensus_queue.cc:1035] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9141bdb017f14ed2925f91ad4b51acd2" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 40607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:21:31.672421 6058 mvcc.cc:205] Tried to move back new op lower bound from 7333065898630451200 to 7333065898516242432. Current Snapshot: MvccSnapshot[applied={T|T < 7333065898630451200}]
I20260925 01:21:31.689690 5568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:21:31.689977 5568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:21:31.693856 5568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6b8741794b4463cafc76d1be6bf117a: Sending DeleteTablet for 3 replicas of tablet 8bf4ff83becf4a0ca5871435d587083a
I20260925 01:21:31.695060 5960 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf4ff83becf4a0ca5871435d587083a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:31 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:52022
I20260925 01:21:31.695369 5826 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf4ff83becf4a0ca5871435d587083a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:31 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:44258
I20260925 01:21:31.695633 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf4ff83becf4a0ca5871435d587083a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:31 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:35572
I20260925 01:21:31.696056 6067 tablet_replica.cc:330] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0: stopping tablet replica
I20260925 01:21:31.696592 6067 raft_consensus.cc:2229] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:21:31.697292 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 5629
I20260925 01:21:31.697363 6067 raft_consensus.cc:2258] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:31.699899 6067 ts_tablet_manager.cc:1905] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:31.703229 6068 tablet_replica.cc:330] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2: stopping tablet replica
I20260925 01:21:31.704259 6069 tablet_replica.cc:330] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd: stopping tablet replica
I20260925 01:21:31.704663 6069 raft_consensus.cc:2229] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:21:31.703704 6068 raft_consensus.cc:2229] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:21:31.705255 6068 raft_consensus.cc:2258] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:31.705271 6069 raft_consensus.cc:2258] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:31.707564 6069 ts_tablet_manager.cc:1905] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:31.708456 6067 ts_tablet_manager.cc:1918] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:31.708704 6067 log.cc:1196] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/8bf4ff83becf4a0ca5871435d587083a
I20260925 01:21:31.709339 6067 ts_tablet_manager.cc:1939] T 8bf4ff83becf4a0ca5871435d587083a P f2505639f3e84a8da5e384a306aa62f0: Deleting consensus metadata
I20260925 01:21:31.711247 5555 catalog_manager.cc:4841] TS f2505639f3e84a8da5e384a306aa62f0 (127.20.102.131:45475): tablet 8bf4ff83becf4a0ca5871435d587083a (table test-table [id=a69eec96ace34961a6bdf5b4e52259b0]) successfully deleted
I20260925 01:21:31.711589 6068 ts_tablet_manager.cc:1905] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:31.716771 6069 ts_tablet_manager.cc:1918] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:31.717034 6069 log.cc:1196] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/8bf4ff83becf4a0ca5871435d587083a
I20260925 01:21:31.717730 6069 ts_tablet_manager.cc:1939] T 8bf4ff83becf4a0ca5871435d587083a P b2c8c39f4e2c4b16a6590f98f2d0d7cd: Deleting consensus metadata
Sep 25 01:21:31 dist-test-slave-s12s krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:21:31.718636 6068 ts_tablet_manager.cc:1918] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:31.718890 6068 log.cc:1196] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/8bf4ff83becf4a0ca5871435d587083a
I20260925 01:21:31.719507 6068 ts_tablet_manager.cc:1939] T 8bf4ff83becf4a0ca5871435d587083a P 9141bdb017f14ed2925f91ad4b51acd2: Deleting consensus metadata
I20260925 01:21:31.719624 5555 catalog_manager.cc:4841] TS b2c8c39f4e2c4b16a6590f98f2d0d7cd (127.20.102.129:44577): tablet 8bf4ff83becf4a0ca5871435d587083a (table test-table [id=a69eec96ace34961a6bdf5b4e52259b0]) successfully deleted
I20260925 01:21:31.721607 5555 catalog_manager.cc:4841] TS 9141bdb017f14ed2925f91ad4b51acd2 (127.20.102.130:40607): tablet 8bf4ff83becf4a0ca5871435d587083a (table test-table [id=a69eec96ace34961a6bdf5b4e52259b0]) successfully deleted
I20260925 01:21:31.725723 6043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.699048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35588 (local address 127.20.102.129:44577)
0925 01:21:31.699284 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:31.699293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:31.703227 (+ 3934us) server_negotiation.cc:407] Connection header received
0925 01:21:31.703527 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:31.703537 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:31.703648 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:31.703774 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:31.705602 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.706816 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.708788 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.709499 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.709615 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.719218 (+ 9603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:31.719237 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:31.719261 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:31.719312 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:31.722811 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:31.723682 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:31.723688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:31.723691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:31.723820 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:31.724237 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:31.724240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:31.724242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:31.724585 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:31.724882 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:31.725223 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 01:21:31.725391 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 01:21:31.726328 5752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:31.833343 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 5629
I20260925 01:21:31.851590 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 5763
Sep 25 01:21:31 dist-test-slave-s12s krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:21:31.869788 6041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.852521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.20.102.130:40607)
0925 01:21:31.852754 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:31.852758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:31.852777 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:21:31.852856 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:31.852861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:31.852921 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:31.853006 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:31.854603 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.855369 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.857450 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.857870 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.857972 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.862999 (+ 5027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:31.863020 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:31.863040 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:31.863089 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:31.866795 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:31.867492 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:31.867502 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:31.867507 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:31.867672 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:31.868021 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:31.868026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:31.868030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:31.868519 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:31.868852 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:31.869178 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 01:21:31.869403 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:21:31.870625 5886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:31.960440 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 5763
I20260925 01:21:31.978925 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 5897
Sep 25 01:21:31 dist-test-slave-s12s krb5kdc[5523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:21:31.995848 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:31.979843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.20.102.131:45475)
0925 01:21:31.980064 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:31.980071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:31.980093 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:21:31.980214 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:31.980219 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:31.980290 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:31.980398 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:31.981844 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.982721 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.983969 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:31.984463 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:31.984612 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:31.989504 (+ 4892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:31.989517 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:31.989532 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:31.989568 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:31.992662 (+ 3094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:31.993577 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:31.993584 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:31.993586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:31.993760 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:31.994186 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:31.994191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:31.994201 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:31.994590 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:31.994816 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:31.995263 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 01:21:31.995451 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:21:31.996644 6020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:32.119630 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 5897
I20260925 01:21:32.139714 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 5536
I20260925 01:21:32.140949 5598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:32.252502 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 5536
2026-09-25T01:21:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4509 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T01:21:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:21:32Z Disabled control of system clock
I20260925 01:21:32.332794 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:37105
--webserver_interface=127.20.102.190
--webserver_port=0
--builtin_ntp_servers=127.20.102.148:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:37105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:21:32.691974 6078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:32.692270 6078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:32.692405 6078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:32.700219 6078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:21:32.700336 6078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:32.700403 6078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:32.700526 6078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:21:32.700582 6078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:21:32.710291 6078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:37105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:37105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:32.711639 6078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:32.713593 6078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:32.726775 6083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:32.726814 6084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:32.726732 6086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:32.727810 6078 server_base.cc:1034] running on GCE node
I20260925 01:21:32.728621 6078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:32.730078 6078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:32.731325 6078 hybrid_clock.cc:648] HybridClock initialized: now 1790299292731257 us; error 60 us; skew 500 ppm
I20260925 01:21:32.731712 6078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:21:32.734269 6078 webserver.cc:460] Webserver started at http://127.20.102.190:41283/ using document root <none> and password file <none>
I20260925 01:21:32.734980 6078 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:32.735116 6078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:32.735457 6078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:32.738528 6078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "ca00f680eafd4a11a156e37f416e1df2"
format_stamp: "Formatted at 2026-09-25 01:21:32 on dist-test-slave-s12s"
server_key: "1253448e6d962cf1a0cadb7c18b11186"
server_key_iv: "3cc8268df0b86f66cfeb1a2265d75436"
server_key_version: "encryptionkey@0"
I20260925 01:21:32.739398 6078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "ca00f680eafd4a11a156e37f416e1df2"
format_stamp: "Formatted at 2026-09-25 01:21:32 on dist-test-slave-s12s"
server_key: "1253448e6d962cf1a0cadb7c18b11186"
server_key_iv: "3cc8268df0b86f66cfeb1a2265d75436"
server_key_version: "encryptionkey@0"
I20260925 01:21:32.744831 6078 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 01:21:32.748346 6092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:32.749930 6078 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:21:32.750154 6078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "ca00f680eafd4a11a156e37f416e1df2"
format_stamp: "Formatted at 2026-09-25 01:21:32 on dist-test-slave-s12s"
server_key: "1253448e6d962cf1a0cadb7c18b11186"
server_key_iv: "3cc8268df0b86f66cfeb1a2265d75436"
server_key_version: "encryptionkey@0"
I20260925 01:21:32.750389 6078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:32.785224 6078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:32.786159 6078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:32.786479 6078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:32.806193 6078 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:37105
I20260925 01:21:32.806212 6143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:37105 every 8 connection(s)
I20260925 01:21:32.807639 6078 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:21:32.811496 6144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:32.815580 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 6078
I20260925 01:21:32.815778 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:21:32.816354 20890 external_mini_cluster.cc:1428] Setting key 38796ea447bc06db8ae0f156329b3bac
I20260925 01:21:32.822965 6144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2: Bootstrap starting.
I20260925 01:21:32.826653 6144 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:32.828042 6144 log.cc:824] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:32.831051 6147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:32.818389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37526 (local address 127.20.102.190:37105)
0925 01:21:32.819398 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:32.819409 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:32.819453 (+ 44us) server_negotiation.cc:407] Connection header received
0925 01:21:32.821308 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:32.821331 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:32.821789 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:32.822120 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:32.822801 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:32.824178 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:32.826169 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:32.826906 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:32.827042 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:32.828372 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:32.828380 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:32.828394 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:32.829240 (+ 846us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:32.829359 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:32.829389 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:32.829520 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:32.829807 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 01:21:32.830149 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":353,"threads_started":1}
I20260925 01:21:32.831300 6144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2: No bootstrap required, opened a new log
I20260925 01:21:32.839197 6144 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca00f680eafd4a11a156e37f416e1df2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37105 } }
I20260925 01:21:32.839584 6144 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:32.839696 6144 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca00f680eafd4a11a156e37f416e1df2, State: Initialized, Role: FOLLOWER
I20260925 01:21:32.840310 6144 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca00f680eafd4a11a156e37f416e1df2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37105 } }
I20260925 01:21:32.840559 6144 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:21:32.840765 6144 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:21:32.840957 6144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:32.843106 6144 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca00f680eafd4a11a156e37f416e1df2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37105 } }
I20260925 01:21:32.843657 6144 leader_election.cc:302] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca00f680eafd4a11a156e37f416e1df2; no voters:
I20260925 01:21:32.844264 6144 leader_election.cc:288] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:21:32.844421 6149 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:32.846055 6149 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [term 1 LEADER]: Becoming Leader. State: Replica: ca00f680eafd4a11a156e37f416e1df2, State: Running, Role: LEADER
I20260925 01:21:32.846506 6144 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:21:32.846529 6149 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca00f680eafd4a11a156e37f416e1df2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37105 } }
I20260925 01:21:32.849527 6151 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca00f680eafd4a11a156e37f416e1df2. Latest consensus state: current_term: 1 leader_uuid: "ca00f680eafd4a11a156e37f416e1df2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca00f680eafd4a11a156e37f416e1df2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37105 } } }
I20260925 01:21:32.850046 6151 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:32.849902 6150 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca00f680eafd4a11a156e37f416e1df2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca00f680eafd4a11a156e37f416e1df2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 37105 } } }
I20260925 01:21:32.850241 6150 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:32.852283 6157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:21:32.857141 6157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:21:32.865243 6157 catalog_manager.cc:1347] Generated new cluster ID: 89412eaad90d41d0803e471c7234bb68
I20260925 01:21:32.865381 6157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:21:32.877422 6157 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:21:32.878793 6157 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:21:32.889079 6157 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2: Generated new TSK 0
I20260925 01:21:32.890007 6157 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:21:32.904168 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37105
--builtin_ntp_servers=127.20.102.148:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:21:33.257586 6168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:33.257870 6168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:33.258045 6168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:33.265539 6168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:33.265641 6168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:33.265847 6168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:21:33.275199 6168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:33.276751 6168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:33.278648 6168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:33.292248 6174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:33.292323 6173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:33.292358 6176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:33.293203 6168 server_base.cc:1034] running on GCE node
I20260925 01:21:33.293840 6168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:33.294842 6168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:33.296082 6168 hybrid_clock.cc:648] HybridClock initialized: now 1790299293296027 us; error 46 us; skew 500 ppm
I20260925 01:21:33.296433 6168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:21:33.298750 6168 webserver.cc:460] Webserver started at http://127.20.102.129:45913/ using document root <none> and password file <none>
I20260925 01:21:33.299468 6168 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:33.299612 6168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:33.299937 6168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:33.302901 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "7adb2da0d0a3435699201b646411f6bc"
format_stamp: "Formatted at 2026-09-25 01:21:33 on dist-test-slave-s12s"
server_key: "3e9157f6aa3ec24852577f8452593681"
server_key_iv: "8b8d17d618cfb9ebf12a95220cdd796a"
server_key_version: "encryptionkey@0"
I20260925 01:21:33.303723 6168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "7adb2da0d0a3435699201b646411f6bc"
format_stamp: "Formatted at 2026-09-25 01:21:33 on dist-test-slave-s12s"
server_key: "3e9157f6aa3ec24852577f8452593681"
server_key_iv: "8b8d17d618cfb9ebf12a95220cdd796a"
server_key_version: "encryptionkey@0"
I20260925 01:21:33.308904 6168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260925 01:21:33.312271 6182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:33.313599 6168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:21:33.313815 6168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "7adb2da0d0a3435699201b646411f6bc"
format_stamp: "Formatted at 2026-09-25 01:21:33 on dist-test-slave-s12s"
server_key: "3e9157f6aa3ec24852577f8452593681"
server_key_iv: "8b8d17d618cfb9ebf12a95220cdd796a"
server_key_version: "encryptionkey@0"
I20260925 01:21:33.314034 6168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:33.328209 6168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:33.329099 6168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:33.329407 6168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:33.330432 6168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:33.332338 6168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:33.332461 6168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:33.332588 6168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:33.332708 6168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:33.365453 6168 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:43551
I20260925 01:21:33.365501 6294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:43551 every 8 connection(s)
I20260925 01:21:33.366832 6168 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:21:33.374360 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 6168
I20260925 01:21:33.374555 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:21:33.374997 20890 external_mini_cluster.cc:1428] Setting key 14bb7ddc8014e862787d55ae78731cab
I20260925 01:21:33.377912 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37105
--builtin_ntp_servers=127.20.102.148:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:21:33.380141 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:33.369559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:45847 (local address 127.20.102.190:37105)
0925 01:21:33.370022 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:33.370026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:33.371245 (+ 1219us) server_negotiation.cc:407] Connection header received
0925 01:21:33.373053 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:33.373057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:33.373128 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:33.373226 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:33.374259 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:33.375095 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:33.377276 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:33.377678 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:33.377816 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:33.379293 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:33.379297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:33.379301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:33.379318 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:33.379366 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:33.379378 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:33.379470 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:33.379699 (+ 229us) server_negotiation.cc:299] Negotiation successful
0925 01:21:33.379795 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":224,"threads_started":1}
I20260925 01:21:33.381183 6298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:33.370213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37105 (local address 127.20.102.129:45847)
0925 01:21:33.371082 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:33.371124 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:33.372813 (+ 1689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:33.373426 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:33.373434 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:33.373519 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:33.374085 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:33.374102 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:33.375271 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:33.375276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:33.377138 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:33.377151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:33.377919 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:33.377927 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:33.378244 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:33.378971 (+ 727us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:21:33.378974 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:33.379013 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:21:33.379095 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:33.379482 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:33.379495 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:33.379504 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 01:21:33.379686 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 01:21:33.380083 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":268,"threads_started":1}
I20260925 01:21:33.382877 6295 heartbeater.cc:347] Connected to a master server at 127.20.102.190:37105
I20260925 01:21:33.383216 6295 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:33.384069 6295 heartbeater.cc:511] Master 127.20.102.190:37105 requested a full tablet report, sending...
I20260925 01:21:33.386317 6109 ts_manager.cc:194] Registered new tserver with Master: 7adb2da0d0a3435699201b646411f6bc (127.20.102.129:43551)
I20260925 01:21:33.387992 6109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.102.129:45847
W20260925 01:21:33.713107 6299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:33.713387 6299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:33.713536 6299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:33.721170 6299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:33.721267 6299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:33.721426 6299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:21:33.731868 6299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:33.733261 6299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:33.735035 6299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:33.749068 6305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:33.749171 6304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:33.749182 6307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:33.750361 6299 server_base.cc:1034] running on GCE node
I20260925 01:21:33.750970 6299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:33.751905 6299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:33.753160 6299 hybrid_clock.cc:648] HybridClock initialized: now 1790299293753084 us; error 73 us; skew 500 ppm
I20260925 01:21:33.753464 6299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:21:33.756070 6299 webserver.cc:460] Webserver started at http://127.20.102.130:43771/ using document root <none> and password file <none>
I20260925 01:21:33.756791 6299 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:33.756888 6299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:33.757157 6299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:33.759935 6299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "8fef28120bd94325a2482d3b91a96989"
format_stamp: "Formatted at 2026-09-25 01:21:33 on dist-test-slave-s12s"
server_key: "93ecbe5e9dab30110b2e62407bd80afa"
server_key_iv: "ab9cafc9fd2b3f8b9309f256d53d06fd"
server_key_version: "encryptionkey@0"
I20260925 01:21:33.760664 6299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "8fef28120bd94325a2482d3b91a96989"
format_stamp: "Formatted at 2026-09-25 01:21:33 on dist-test-slave-s12s"
server_key: "93ecbe5e9dab30110b2e62407bd80afa"
server_key_iv: "ab9cafc9fd2b3f8b9309f256d53d06fd"
server_key_version: "encryptionkey@0"
I20260925 01:21:33.765934 6299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 01:21:33.769495 6313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:33.770887 6299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:21:33.771057 6299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "8fef28120bd94325a2482d3b91a96989"
format_stamp: "Formatted at 2026-09-25 01:21:33 on dist-test-slave-s12s"
server_key: "93ecbe5e9dab30110b2e62407bd80afa"
server_key_iv: "ab9cafc9fd2b3f8b9309f256d53d06fd"
server_key_version: "encryptionkey@0"
I20260925 01:21:33.771277 6299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:33.790833 6299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:33.791762 6299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:33.792105 6299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:33.793133 6299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:33.794763 6299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:33.794894 6299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:33.795025 6299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:33.795121 6299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:33.831532 6299 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:34613
I20260925 01:21:33.831553 6425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:34613 every 8 connection(s)
I20260925 01:21:33.832945 6299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:21:33.834571 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 6299
I20260925 01:21:33.834775 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:21:33.835265 20890 external_mini_cluster.cc:1428] Setting key b9c69474b7811a3b2104486a51f220d0
I20260925 01:21:33.838436 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37105
--builtin_ntp_servers=127.20.102.148:44593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:21:33.849937 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:33.838275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:46077 (local address 127.20.102.190:37105)
0925 01:21:33.838461 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:33.838467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:33.839159 (+ 692us) server_negotiation.cc:407] Connection header received
0925 01:21:33.842032 (+ 2873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:33.842036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:33.842116 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:33.842203 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:33.843691 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:33.844416 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:33.846925 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:33.847374 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:33.847520 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:33.849010 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:33.849019 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:33.849027 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:33.849061 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:33.849102 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:33.849120 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:33.849215 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:33.849426 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 01:21:33.849540 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:21:33.850853 6428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:33.837877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37105 (local address 127.20.102.130:46077)
0925 01:21:33.838972 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:33.839021 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:33.841735 (+ 2714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:33.842464 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:33.842477 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:33.842602 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:33.843492 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:33.843518 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:33.844572 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:33.844579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:33.846781 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:33.846799 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:33.847590 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:33.847601 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:33.847909 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:33.848633 (+ 724us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:21:33.848635 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:33.848721 (+ 86us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:21:33.848825 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:33.849220 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:33.849228 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:33.849233 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 01:21:33.849384 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 01:21:33.849684 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":388,"threads_started":1}
I20260925 01:21:33.852389 6426 heartbeater.cc:347] Connected to a master server at 127.20.102.190:37105
I20260925 01:21:33.852743 6426 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:33.853597 6426 heartbeater.cc:511] Master 127.20.102.190:37105 requested a full tablet report, sending...
I20260925 01:21:33.855073 6109 ts_manager.cc:194] Registered new tserver with Master: 8fef28120bd94325a2482d3b91a96989 (127.20.102.130:34613)
I20260925 01:21:33.855934 6109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.102.130:46077
W20260925 01:21:34.171202 6429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:34.171481 6429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:34.171640 6429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:34.179049 6429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:34.179145 6429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:34.179318 6429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:21:34.188251 6429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:44593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:37105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:34.189603 6429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:34.191342 6429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:34.204118 6434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:34.204087 6437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:34.204341 6435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:34.205405 6429 server_base.cc:1034] running on GCE node
I20260925 01:21:34.206071 6429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:34.207057 6429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:34.208304 6429 hybrid_clock.cc:648] HybridClock initialized: now 1790299294208247 us; error 50 us; skew 500 ppm
I20260925 01:21:34.208716 6429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:21:34.211113 6429 webserver.cc:460] Webserver started at http://127.20.102.131:41651/ using document root <none> and password file <none>
I20260925 01:21:34.211843 6429 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:34.211978 6429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:34.212302 6429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:34.215273 6429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "8ad2b7fb261d48868b0e469b0034562b"
format_stamp: "Formatted at 2026-09-25 01:21:34 on dist-test-slave-s12s"
server_key: "4d14fe8f9d08ea0d178719bf421609ed"
server_key_iv: "f23e5daf95a7f01e4c81c69541bda684"
server_key_version: "encryptionkey@0"
I20260925 01:21:34.216143 6429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "8ad2b7fb261d48868b0e469b0034562b"
format_stamp: "Formatted at 2026-09-25 01:21:34 on dist-test-slave-s12s"
server_key: "4d14fe8f9d08ea0d178719bf421609ed"
server_key_iv: "f23e5daf95a7f01e4c81c69541bda684"
server_key_version: "encryptionkey@0"
I20260925 01:21:34.221796 6429 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260925 01:21:34.225322 6443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:34.226807 6429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:21:34.226989 6429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "8ad2b7fb261d48868b0e469b0034562b"
format_stamp: "Formatted at 2026-09-25 01:21:34 on dist-test-slave-s12s"
server_key: "4d14fe8f9d08ea0d178719bf421609ed"
server_key_iv: "f23e5daf95a7f01e4c81c69541bda684"
server_key_version: "encryptionkey@0"
I20260925 01:21:34.227207 6429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:34.243022 6429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:34.243862 6429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:34.244181 6429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:34.245236 6429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:34.246833 6429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:34.246961 6429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:34.247092 6429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:34.247195 6429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:34.282631 6429 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:38107
I20260925 01:21:34.282694 6555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:38107 every 8 connection(s)
I20260925 01:21:34.284111 6429 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:21:34.287431 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 6429
I20260925 01:21:34.287638 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:21:34.288313 20890 external_mini_cluster.cc:1428] Setting key 673ed4a5b722c0273dad3395683c23c7
I20260925 01:21:34.298202 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.287097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:41855 (local address 127.20.102.190:37105)
0925 01:21:34.287393 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:34.287398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:34.289044 (+ 1646us) server_negotiation.cc:407] Connection header received
0925 01:21:34.291719 (+ 2675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:34.291725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:34.291833 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:34.291916 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:34.293167 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.293914 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.295564 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.295984 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.296079 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.297457 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:34.297462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:34.297465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:34.297483 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:34.297513 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:34.297525 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:34.297599 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:34.297875 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 01:21:34.297952 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20260925 01:21:34.299283 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.287569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:37105 (local address 127.20.102.131:41855)
0925 01:21:34.288821 (+ 1252us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:34.288881 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:34.291349 (+ 2468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:34.292153 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:34.292162 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:34.292254 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:34.293006 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:34.293026 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.294071 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.294075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:34.295427 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:34.295439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.296174 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.296179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:34.296375 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.297150 (+ 775us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:21:34.297153 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:34.297191 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:21:34.297266 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:34.297655 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:34.297664 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:34.297672 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 01:21:34.297842 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 01:21:34.298158 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":338,"threads_started":1}
I20260925 01:21:34.300745 6556 heartbeater.cc:347] Connected to a master server at 127.20.102.190:37105
I20260925 01:21:34.301051 6556 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:34.301821 6556 heartbeater.cc:511] Master 127.20.102.190:37105 requested a full tablet report, sending...
I20260925 01:21:34.303463 6109 ts_manager.cc:194] Registered new tserver with Master: 8ad2b7fb261d48868b0e469b0034562b (127.20.102.131:38107)
I20260925 01:21:34.304219 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:21:34.304483 6109 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.102.131:41855
I20260925 01:21:34.314174 6297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.308420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37532 (local address 127.20.102.190:37105)
0925 01:21:34.308641 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:34.308646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:34.309130 (+ 484us) server_negotiation.cc:407] Connection header received
0925 01:21:34.309265 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:34.309268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:34.309338 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:34.309418 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:34.309872 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.310535 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.311839 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.312255 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.312345 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.313462 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:34.313467 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:34.313471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:34.313489 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:34.313530 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:34.313543 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:34.313641 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:34.313830 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 01:21:34.313910 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:21:34.318588 6109 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:21:34.326383 6109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:21:34.348815 6568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.339150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:34613 (local address 127.0.0.1:54018)
0925 01:21:34.339869 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:34.339908 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:34.340065 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:34.341226 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:34.341234 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:34.341270 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:34.341707 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:34.341728 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.343804 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.343812 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:34.346109 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:34.346132 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.347744 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.347750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:34.348048 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.348125 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 01:21:34.348243 (+ 118us) client_negotiation.cc:240] Negotiation successful
0925 01:21:34.348398 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":41,"thread_start_us":305,"threads_started":1}
I20260925 01:21:34.349654 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.339265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.20.102.130:34613)
0925 01:21:34.339892 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:34.339902 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:34.340022 (+ 120us) server_negotiation.cc:407] Connection header received
0925 01:21:34.340290 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:34.340301 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:34.340604 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:34.340872 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:34.342007 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.343655 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.346332 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.347550 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.347774 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.348039 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:34.349081 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0925 01:21:34.349262 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":329,"threads_started":1}
I20260925 01:21:34.350803 6570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.340414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:38107 (local address 127.0.0.1:38530)
0925 01:21:34.340997 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:34.341018 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:34.341142 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:34.342361 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:34.342368 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:34.342400 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:34.342819 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:34.342831 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.345038 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.345044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:34.347415 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:34.347430 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.349851 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.349859 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:34.350107 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.350146 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 01:21:34.350222 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 01:21:34.350374 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":4,"thread_start_us":288,"threads_started":1}
I20260925 01:21:34.350921 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.341074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43551 (local address 127.0.0.1:48600)
0925 01:21:34.342025 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:34.342042 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:34.342154 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:34.343284 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:34.343288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:34.343314 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:34.343687 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:34.343699 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.345811 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.345816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:34.348047 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:34.348062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.350132 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.350139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:34.350326 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.350361 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:21:34.350421 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 01:21:34.350565 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":29,"thread_start_us":750,"threads_started":1}
I20260925 01:21:34.352427 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.341602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48600 (local address 127.20.102.129:43551)
0925 01:21:34.342689 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:34.342699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:34.342732 (+ 33us) server_negotiation.cc:407] Connection header received
0925 01:21:34.342827 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:34.342833 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:34.343049 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:34.343910 (+ 861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:34.344138 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.345648 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.348264 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.349900 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.351078 (+ 1178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.351763 (+ 685us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:34.351852 (+ 89us) server_negotiation.cc:299] Negotiation successful
0925 01:21:34.352042 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913,"thread_start_us":1198,"threads_started":1}
I20260925 01:21:34.352648 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.340797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.20.102.131:38107)
0925 01:21:34.341770 (+ 973us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:34.341780 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:34.341815 (+ 35us) server_negotiation.cc:407] Connection header received
0925 01:21:34.341910 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:34.341917 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:34.342152 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:34.342995 (+ 843us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:34.343283 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.344882 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.347668 (+ 2786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.349593 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.351761 (+ 2168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.351985 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:34.352063 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 01:21:34.352238 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":617,"threads_started":1}
I20260925 01:21:34.353788 6361 tablet_service.cc:1467] Processing CreateTablet for tablet 202c2d0aa94548bea1e0e068711d1cda (DEFAULT_TABLE table=test-table [id=4be367bad10c4776ac6189395861c8ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:34.355376 6361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202c2d0aa94548bea1e0e068711d1cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:34.356184 6491 tablet_service.cc:1467] Processing CreateTablet for tablet 202c2d0aa94548bea1e0e068711d1cda (DEFAULT_TABLE table=test-table [id=4be367bad10c4776ac6189395861c8ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:34.356249 6230 tablet_service.cc:1467] Processing CreateTablet for tablet 202c2d0aa94548bea1e0e068711d1cda (DEFAULT_TABLE table=test-table [id=4be367bad10c4776ac6189395861c8ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:34.357509 6491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202c2d0aa94548bea1e0e068711d1cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:34.357587 6230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202c2d0aa94548bea1e0e068711d1cda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:34.376710 6574 tablet_bootstrap.cc:492] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989: Bootstrap starting.
I20260925 01:21:34.377068 6575 tablet_bootstrap.cc:492] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b: Bootstrap starting.
I20260925 01:21:34.381439 6576 tablet_bootstrap.cc:492] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc: Bootstrap starting.
I20260925 01:21:34.382160 6575 tablet_bootstrap.cc:654] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:34.383579 6575 log.cc:824] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:34.383905 6574 tablet_bootstrap.cc:654] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:34.385800 6574 log.cc:824] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:34.387243 6575 tablet_bootstrap.cc:492] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b: No bootstrap required, opened a new log
I20260925 01:21:34.387363 6576 tablet_bootstrap.cc:654] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:34.387578 6575 ts_tablet_manager.cc:1397] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260925 01:21:34.388339 6576 log.cc:824] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:34.389233 6574 tablet_bootstrap.cc:492] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989: No bootstrap required, opened a new log
I20260925 01:21:34.389528 6574 ts_tablet_manager.cc:1397] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260925 01:21:34.390774 6576 tablet_bootstrap.cc:492] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc: No bootstrap required, opened a new log
I20260925 01:21:34.391028 6576 ts_tablet_manager.cc:1397] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260925 01:21:34.391990 6295 heartbeater.cc:503] Master 127.20.102.190:37105 was elected leader, sending a full tablet report...
I20260925 01:21:34.400413 6574 raft_consensus.cc:348] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fef28120bd94325a2482d3b91a96989" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34613 } } peers { permanent_uuid: "8ad2b7fb261d48868b0e469b0034562b" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38107 } } peers { permanent_uuid: "7adb2da0d0a3435699201b646411f6bc" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43551 } }
I20260925 01:21:34.400442 6576 raft_consensus.cc:348] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fef28120bd94325a2482d3b91a96989" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34613 } } peers { permanent_uuid: "8ad2b7fb261d48868b0e469b0034562b" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38107 } } peers { permanent_uuid: "7adb2da0d0a3435699201b646411f6bc" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43551 } }
I20260925 01:21:34.400924 6576 raft_consensus.cc:374] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:34.400935 6574 raft_consensus.cc:374] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:34.401031 6574 raft_consensus.cc:729] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fef28120bd94325a2482d3b91a96989, State: Initialized, Role: FOLLOWER
I20260925 01:21:34.401054 6576 raft_consensus.cc:729] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7adb2da0d0a3435699201b646411f6bc, State: Initialized, Role: FOLLOWER
I20260925 01:21:34.401721 6574 consensus_queue.cc:260] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fef28120bd94325a2482d3b91a96989" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34613 } } peers { permanent_uuid: "8ad2b7fb261d48868b0e469b0034562b" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38107 } } peers { permanent_uuid: "7adb2da0d0a3435699201b646411f6bc" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43551 } }
I20260925 01:21:34.401705 6576 consensus_queue.cc:260] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fef28120bd94325a2482d3b91a96989" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34613 } } peers { permanent_uuid: "8ad2b7fb261d48868b0e469b0034562b" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38107 } } peers { permanent_uuid: "7adb2da0d0a3435699201b646411f6bc" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43551 } }
I20260925 01:21:34.402380 6575 raft_consensus.cc:348] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fef28120bd94325a2482d3b91a96989" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34613 } } peers { permanent_uuid: "8ad2b7fb261d48868b0e469b0034562b" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38107 } } peers { permanent_uuid: "7adb2da0d0a3435699201b646411f6bc" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43551 } }
I20260925 01:21:34.402872 6575 raft_consensus.cc:374] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:34.403004 6575 raft_consensus.cc:729] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ad2b7fb261d48868b0e469b0034562b, State: Initialized, Role: FOLLOWER
I20260925 01:21:34.403755 6576 ts_tablet_manager.cc:1428] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260925 01:21:34.403755 6575 consensus_queue.cc:260] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fef28120bd94325a2482d3b91a96989" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34613 } } peers { permanent_uuid: "8ad2b7fb261d48868b0e469b0034562b" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38107 } } peers { permanent_uuid: "7adb2da0d0a3435699201b646411f6bc" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43551 } }
I20260925 01:21:34.408430 6574 ts_tablet_manager.cc:1428] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989: Time spent starting tablet: real 0.019s user 0.011s sys 0.007s
I20260925 01:21:34.408720 6426 heartbeater.cc:503] Master 127.20.102.190:37105 was elected leader, sending a full tablet report...
I20260925 01:21:34.409503 6556 heartbeater.cc:503] Master 127.20.102.190:37105 was elected leader, sending a full tablet report...
I20260925 01:21:34.409958 6575 ts_tablet_manager.cc:1428] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b: Time spent starting tablet: real 0.022s user 0.010s sys 0.010s
I20260925 01:21:34.453186 6582 raft_consensus.cc:482] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:21:34.453632 6582 raft_consensus.cc:504] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fef28120bd94325a2482d3b91a96989" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34613 } } peers { permanent_uuid: "8ad2b7fb261d48868b0e469b0034562b" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38107 } } peers { permanent_uuid: "7adb2da0d0a3435699201b646411f6bc" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43551 } }
I20260925 01:21:34.455633 6582 leader_election.cc:288] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8fef28120bd94325a2482d3b91a96989 (127.20.102.130:34613), 7adb2da0d0a3435699201b646411f6bc (127.20.102.129:43551)
I20260925 01:21:34.461403 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.455978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:34613 (local address 127.20.102.131:33645)
0925 01:21:34.456167 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:34.456188 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:34.456303 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:34.456955 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:34.456960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:34.456994 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:34.457178 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:34.457186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.458210 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.458218 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:34.459523 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:34.459535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.460599 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.460604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:34.460821 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.460845 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:21:34.460949 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 01:21:34.461102 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 01:21:34.461395 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.456379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:33645 (local address 127.20.102.130:34613)
0925 01:21:34.456560 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:34.456566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:34.456585 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:21:34.456698 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:34.456704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:34.456782 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:34.456910 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:34.457359 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.458080 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.459710 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.460441 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.460607 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.460729 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:34.460980 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 01:21:34.461125 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 01:21:34.462270 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.456583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:36869 (local address 127.20.102.129:43551)
0925 01:21:34.456782 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:34.456788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:34.457117 (+ 329us) server_negotiation.cc:407] Connection header received
0925 01:21:34.457301 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:34.457306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:34.457391 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:34.457487 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:34.458108 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.458780 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.460470 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.461319 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.461481 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.461545 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:34.461876 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 01:21:34.462018 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:21:34.462313 6583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.456335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43551 (local address 127.20.102.131:36869)
0925 01:21:34.456994 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:34.457012 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:34.457156 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:34.457522 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:34.457526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:34.457547 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:34.457917 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:34.457926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.458929 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.458935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:34.460286 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:34.460297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.461474 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.461479 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:34.461654 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.461700 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 01:21:34.461821 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 01:21:34.461971 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"mutex_wait_us":3,"thread_start_us":372,"threads_started":1}
I20260925 01:21:34.462618 6381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202c2d0aa94548bea1e0e068711d1cda" candidate_uuid: "8ad2b7fb261d48868b0e469b0034562b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fef28120bd94325a2482d3b91a96989" is_pre_election: true
I20260925 01:21:34.463150 6381 raft_consensus.cc:2454] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad2b7fb261d48868b0e469b0034562b in term 0.
I20260925 01:21:34.463282 6250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202c2d0aa94548bea1e0e068711d1cda" candidate_uuid: "8ad2b7fb261d48868b0e469b0034562b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7adb2da0d0a3435699201b646411f6bc" is_pre_election: true
I20260925 01:21:34.463851 6250 raft_consensus.cc:2454] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ad2b7fb261d48868b0e469b0034562b in term 0.
I20260925 01:21:34.463851 6445 leader_election.cc:302] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ad2b7fb261d48868b0e469b0034562b, 8fef28120bd94325a2482d3b91a96989; no voters:
I20260925 01:21:34.464268 6582 raft_consensus.cc:2781] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:21:34.464433 6582 raft_consensus.cc:482] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:21:34.464562 6582 raft_consensus.cc:3037] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:34.467084 6582 raft_consensus.cc:504] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fef28120bd94325a2482d3b91a96989" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34613 } } peers { permanent_uuid: "8ad2b7fb261d48868b0e469b0034562b" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38107 } } peers { permanent_uuid: "7adb2da0d0a3435699201b646411f6bc" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43551 } }
I20260925 01:21:34.467702 6582 leader_election.cc:288] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [CANDIDATE]: Term 1 election: Requested vote from peers 8fef28120bd94325a2482d3b91a96989 (127.20.102.130:34613), 7adb2da0d0a3435699201b646411f6bc (127.20.102.129:43551)
I20260925 01:21:34.468468 6381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202c2d0aa94548bea1e0e068711d1cda" candidate_uuid: "8ad2b7fb261d48868b0e469b0034562b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8fef28120bd94325a2482d3b91a96989"
I20260925 01:21:34.468575 6250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202c2d0aa94548bea1e0e068711d1cda" candidate_uuid: "8ad2b7fb261d48868b0e469b0034562b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7adb2da0d0a3435699201b646411f6bc"
I20260925 01:21:34.468834 6250 raft_consensus.cc:3037] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:34.468835 6381 raft_consensus.cc:3037] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:34.472714 6250 raft_consensus.cc:2454] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad2b7fb261d48868b0e469b0034562b in term 1.
I20260925 01:21:34.472714 6381 raft_consensus.cc:2454] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ad2b7fb261d48868b0e469b0034562b in term 1.
I20260925 01:21:34.473163 6445 leader_election.cc:302] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7adb2da0d0a3435699201b646411f6bc, 8ad2b7fb261d48868b0e469b0034562b; no voters:
I20260925 01:21:34.473457 6582 raft_consensus.cc:2781] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:34.473978 6582 raft_consensus.cc:686] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [term 1 LEADER]: Becoming Leader. State: Replica: 8ad2b7fb261d48868b0e469b0034562b, State: Running, Role: LEADER
I20260925 01:21:34.474586 6582 consensus_queue.cc:237] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fef28120bd94325a2482d3b91a96989" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34613 } } peers { permanent_uuid: "8ad2b7fb261d48868b0e469b0034562b" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38107 } } peers { permanent_uuid: "7adb2da0d0a3435699201b646411f6bc" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43551 } }
I20260925 01:21:34.479430 6108 catalog_manager.cc:5495] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ad2b7fb261d48868b0e469b0034562b (127.20.102.131). New cstate: current_term: 1 leader_uuid: "8ad2b7fb261d48868b0e469b0034562b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8fef28120bd94325a2482d3b91a96989" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ad2b7fb261d48868b0e469b0034562b" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7adb2da0d0a3435699201b646411f6bc" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43551 } health_report { overall_health: UNKNOWN } } }
I20260925 01:21:34.521955 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.516234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.20.102.131:38107)
0925 01:21:34.516430 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:34.516434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:34.516452 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:21:34.516560 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:34.516564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:34.516643 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:34.516838 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:21:34.517178 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.517897 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.519487 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.520021 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.520127 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.520571 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:21:34.521172 (+ 601us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:21:34.521288 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:34.521481 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 01:21:34.521574 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 01:21:34.530683 6381 raft_consensus.cc:1261] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989 [term 1 FOLLOWER]: Refusing update from remote peer 8ad2b7fb261d48868b0e469b0034562b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:21:34.530697 6250 raft_consensus.cc:1261] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc [term 1 FOLLOWER]: Refusing update from remote peer 8ad2b7fb261d48868b0e469b0034562b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:21:34.531849 6584 consensus_queue.cc:1035] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7adb2da0d0a3435699201b646411f6bc" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:21:34.532306 6582 consensus_queue.cc:1035] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fef28120bd94325a2482d3b91a96989" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 34613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:21:34.536098 6557 tablet.cc:2369] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:21:34.544838 6587 mvcc.cc:205] Tried to move back new op lower bound from 7333065910386532352 to 7333065910172581888. Current Snapshot: MvccSnapshot[applied={T|T < 7333065910386532352}]
I20260925 01:21:34.545702 6588 mvcc.cc:205] Tried to move back new op lower bound from 7333065910386532352 to 7333065910172581888. Current Snapshot: MvccSnapshot[applied={T|T < 7333065910386532352}]
I20260925 01:21:34.564934 6109 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:21:34.565253 6109 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:21:34.569456 6109 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ca00f680eafd4a11a156e37f416e1df2: Sending DeleteTablet for 3 replicas of tablet 202c2d0aa94548bea1e0e068711d1cda
I20260925 01:21:34.570678 6361 tablet_service.cc:1514] Processing DeleteTablet for tablet 202c2d0aa94548bea1e0e068711d1cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:34 UTC) from {username='slave'} at 127.0.0.1:54018
I20260925 01:21:34.570883 6491 tablet_service.cc:1514] Processing DeleteTablet for tablet 202c2d0aa94548bea1e0e068711d1cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:34 UTC) from {username='slave'} at 127.0.0.1:38530
I20260925 01:21:34.571050 6230 tablet_service.cc:1514] Processing DeleteTablet for tablet 202c2d0aa94548bea1e0e068711d1cda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:34 UTC) from {username='slave'} at 127.0.0.1:48600
I20260925 01:21:34.573168 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 6168
I20260925 01:21:34.582931 6599 tablet_replica.cc:330] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc: stopping tablet replica
I20260925 01:21:34.583212 6597 tablet_replica.cc:330] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989: stopping tablet replica
I20260925 01:21:34.583518 6599 raft_consensus.cc:2229] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:21:34.584193 6599 raft_consensus.cc:2258] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:34.584784 6597 raft_consensus.cc:2229] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:21:34.585345 6598 tablet_replica.cc:330] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b: stopping tablet replica
I20260925 01:21:34.585502 6597 raft_consensus.cc:2258] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:34.585811 6598 raft_consensus.cc:2229] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:21:34.586678 6598 raft_consensus.cc:2258] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:34.588153 6597 ts_tablet_manager.cc:1905] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:34.589218 6598 ts_tablet_manager.cc:1905] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:34.596249 6597 ts_tablet_manager.cc:1918] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:34.596580 6597 log.cc:1196] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/202c2d0aa94548bea1e0e068711d1cda
I20260925 01:21:34.597267 6597 ts_tablet_manager.cc:1939] T 202c2d0aa94548bea1e0e068711d1cda P 8fef28120bd94325a2482d3b91a96989: Deleting consensus metadata
I20260925 01:21:34.597286 6599 ts_tablet_manager.cc:1905] T 202c2d0aa94548bea1e0e068711d1cda P 7adb2da0d0a3435699201b646411f6bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:34.597976 6598 ts_tablet_manager.cc:1918] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:34.598240 6598 log.cc:1196] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/202c2d0aa94548bea1e0e068711d1cda
I20260925 01:21:34.599270 6598 ts_tablet_manager.cc:1939] T 202c2d0aa94548bea1e0e068711d1cda P 8ad2b7fb261d48868b0e469b0034562b: Deleting consensus metadata
I20260925 01:21:34.599510 6094 catalog_manager.cc:4841] TS 8fef28120bd94325a2482d3b91a96989 (127.20.102.130:34613): tablet 202c2d0aa94548bea1e0e068711d1cda (table test-table [id=4be367bad10c4776ac6189395861c8ad]) successfully deleted
I20260925 01:21:34.599833 6573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.574750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48608 (local address 127.20.102.129:43551)
0925 01:21:34.575042 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:34.575052 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:34.588248 (+ 13196us) server_negotiation.cc:407] Connection header received
0925 01:21:34.588807 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:34.588813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:34.588925 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:34.589006 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:34.589703 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.590909 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.592944 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.593809 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.593932 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.597951 (+ 4019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:34.597964 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:34.597984 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:34.598813 (+ 829us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:34.598899 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:34.598938 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:34.599033 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:34.599305 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 01:21:34.599442 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:21:34.601052 6290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:34.601774 6094 catalog_manager.cc:4841] TS 8ad2b7fb261d48868b0e469b0034562b (127.20.102.131:38107): tablet 202c2d0aa94548bea1e0e068711d1cda (table test-table [id=4be367bad10c4776ac6189395861c8ad]) successfully deleted
I20260925 01:21:34.713989 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 6168
I20260925 01:21:34.734035 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 6299
W20260925 01:21:34.734138 6094 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.102.129:43551 (error 108)
I20260925 01:21:34.736517 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.736051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43551 (local address 127.0.0.1:48624)
0925 01:21:34.736223 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:34.736342 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 01:21:34.737614 6094 catalog_manager.cc:4584] TS 7adb2da0d0a3435699201b646411f6bc (127.20.102.129:43551): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 202c2d0aa94548bea1e0e068711d1cda: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43551: connect: Connection refused (error 111)
I20260925 01:21:34.742295 6569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.734914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.20.102.130:34613)
0925 01:21:34.735578 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:34.735585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:34.735603 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:21:34.735669 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:34.735675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:34.735798 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:34.735898 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:34.736523 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.737369 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.739279 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.739685 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.739779 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.740961 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:34.740970 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:34.740984 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:34.741568 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:34.741615 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:34.741634 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:34.741720 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:34.741878 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 01:21:34.741963 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514}
I20260925 01:21:34.742982 6421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:34.791621 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.791002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43551 (local address 127.0.0.1:48636)
0925 01:21:34.791261 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:34.791390 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 01:21:34.793112 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.792699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43551 (local address 127.0.0.1:48638)
0925 01:21:34.792871 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:34.792961 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 01:21:34.835165 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 6299
I20260925 01:21:34.853595 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.853149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43551 (local address 127.0.0.1:48648)
0925 01:21:34.853349 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:34.853439 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 01:21:34.854686 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.854311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43551 (local address 127.0.0.1:48664)
0925 01:21:34.854499 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:34.854565 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260925 01:21:34.859169 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 6429
I20260925 01:21:34.867810 6571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.860135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.20.102.131:38107)
0925 01:21:34.860297 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:34.860301 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:34.860319 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:21:34.860454 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:34.860459 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:34.860562 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:34.860651 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:34.861341 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.862139 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.864201 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:34.864670 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:34.864979 (+ 309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:34.866312 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:34.866321 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:34.866336 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:34.866976 (+ 640us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:34.867027 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:34.867049 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:34.867141 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:34.867331 (+ 190us) server_negotiation.cc:299] Negotiation successful
0925 01:21:34.867411 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:21:34.868618 6551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:34.935271 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.934757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43551 (local address 127.0.0.1:48676)
0925 01:21:34.934968 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:34.935076 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 01:21:34.936599 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:34.936160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43551 (local address 127.0.0.1:48686)
0925 01:21:34.936399 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:34.936475 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260925 01:21:34.991883 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 6429
I20260925 01:21:35.012152 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 6078
I20260925 01:21:35.013476 6139 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:35.124114 6572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:35.123387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43551 (local address 127.0.0.1:48694)
0925 01:21:35.123647 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:35.123850 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43551: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 01:21:35.124398 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 6078
2026-09-25T01:21:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2876 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:21:35 dist-test-slave-s12s krb5kdc[6608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:21:35 dist-test-slave-s12s krb5kdc[6608](info): set up 2 sockets
Sep 25 01:21:35 dist-test-slave-s12s krb5kdc[6608](info): commencing operation
krb5kdc: starting...
W20260925 01:21:36.349082 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.177s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:21:36 dist-test-slave-s12s krb5kdc[6608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:21:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:21:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:21:36.610244 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:34647
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:46617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:34647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:36.950186 6621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:36.950496 6621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:36.950680 6621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:36.959039 6621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:21:36.959187 6621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:36.959270 6621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:36.959358 6621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:21:36.959447 6621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:21:36.969576 6621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:46617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:34647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:34647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:36.970979 6621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:36.972355 6621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:36.984638 6626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:36.984638 6629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:36.984699 6627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:36.990275 6621 server_base.cc:1034] running on GCE node
I20260925 01:21:36.991104 6621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:36.992563 6621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:36.993824 6621 hybrid_clock.cc:648] HybridClock initialized: now 1790299296993772 us; error 42 us; skew 500 ppm
I20260925 01:21:36.994212 6621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:36 dist-test-slave-s12s krb5kdc[6608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299296, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:36.997900 6621 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:21:36.999390 6621 webserver.cc:460] Webserver started at http://127.20.102.190:35669/ using document root <none> and password file <none>
I20260925 01:21:37.000134 6621 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:37.000269 6621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:37.000597 6621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:37.003700 6621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "e91024eabd9843eda595e78c733a9f53"
format_stamp: "Formatted at 2026-09-25 01:21:36 on dist-test-slave-s12s"
server_key: "becc13891f454957fdcb6978c46902a2"
server_key_iv: "5c2e728cbd849e3441f3737679b59ffd"
server_key_version: "encryptionkey@0"
I20260925 01:21:37.004493 6621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "e91024eabd9843eda595e78c733a9f53"
format_stamp: "Formatted at 2026-09-25 01:21:36 on dist-test-slave-s12s"
server_key: "becc13891f454957fdcb6978c46902a2"
server_key_iv: "5c2e728cbd849e3441f3737679b59ffd"
server_key_version: "encryptionkey@0"
I20260925 01:21:37.009800 6621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 01:21:37.013278 6636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:37.014635 6621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 01:21:37.014806 6621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "e91024eabd9843eda595e78c733a9f53"
format_stamp: "Formatted at 2026-09-25 01:21:36 on dist-test-slave-s12s"
server_key: "becc13891f454957fdcb6978c46902a2"
server_key_iv: "5c2e728cbd849e3441f3737679b59ffd"
server_key_version: "encryptionkey@0"
I20260925 01:21:37.015003 6621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:37.025851 6621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:37.029873 6621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:37.030138 6621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:37.049063 6621 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:34647
I20260925 01:21:37.049095 6687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:34647 every 8 connection(s)
I20260925 01:21:37.050570 6621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:21:37.054451 6688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:37.055809 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 6621
I20260925 01:21:37.056005 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:21:37.056455 20890 external_mini_cluster.cc:1428] Setting key 94e639a3356f637dd7e14352ee432888
I20260925 01:21:37.068917 6688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53: Bootstrap starting.
Sep 25 01:21:37 dist-test-slave-s12s krb5kdc[6608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:21:37.072086 6688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:37.073295 6688 log.cc:824] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:37.076201 6688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53: No bootstrap required, opened a new log
I20260925 01:21:37.079779 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:37.058389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53230 (local address 127.20.102.190:34647)
0925 01:21:37.059240 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:37.059260 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:37.059314 (+ 54us) server_negotiation.cc:407] Connection header received
0925 01:21:37.061181 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:37.061211 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:37.061728 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:37.062175 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:37.063497 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:37.065585 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:37.066951 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:37.068013 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:37.068208 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:37.072402 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:37.072410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:37.072427 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:37.072471 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:37.075664 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:37.076344 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:37.076351 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:37.076357 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:37.076518 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:37.076947 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:37.076952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:37.076957 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:37.077476 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:37.077809 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:37.078310 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 01:21:37.078767 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":369,"threads_started":1}
I20260925 01:21:37.084949 6688 raft_consensus.cc:348] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91024eabd9843eda595e78c733a9f53" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34647 } }
I20260925 01:21:37.085341 6688 raft_consensus.cc:374] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:37.085449 6688 raft_consensus.cc:729] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e91024eabd9843eda595e78c733a9f53, State: Initialized, Role: FOLLOWER
I20260925 01:21:37.085981 6688 consensus_queue.cc:260] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91024eabd9843eda595e78c733a9f53" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34647 } }
I20260925 01:21:37.086225 6688 raft_consensus.cc:388] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:21:37.086364 6688 raft_consensus.cc:482] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:21:37.086530 6688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:37.089154 6688 raft_consensus.cc:504] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91024eabd9843eda595e78c733a9f53" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34647 } }
I20260925 01:21:37.089705 6688 leader_election.cc:302] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e91024eabd9843eda595e78c733a9f53; no voters:
I20260925 01:21:37.090277 6688 leader_election.cc:288] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:21:37.090431 6693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:37.091128 6693 raft_consensus.cc:686] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [term 1 LEADER]: Becoming Leader. State: Replica: e91024eabd9843eda595e78c733a9f53, State: Running, Role: LEADER
I20260925 01:21:37.091691 6693 consensus_queue.cc:237] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91024eabd9843eda595e78c733a9f53" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34647 } }
I20260925 01:21:37.092391 6688 sys_catalog.cc:564] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:21:37.094820 6694 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e91024eabd9843eda595e78c733a9f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91024eabd9843eda595e78c733a9f53" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34647 } } }
I20260925 01:21:37.095661 6694 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:37.094959 6695 sys_catalog.cc:455] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e91024eabd9843eda595e78c733a9f53. Latest consensus state: current_term: 1 leader_uuid: "e91024eabd9843eda595e78c733a9f53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e91024eabd9843eda595e78c733a9f53" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 34647 } } }
I20260925 01:21:37.096096 6695 sys_catalog.cc:458] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:37.097612 6702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:21:37.103159 6702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:21:37.112545 6702 catalog_manager.cc:1347] Generated new cluster ID: 9880547b773a42c6838f3f23d751bb3d
I20260925 01:21:37.112690 6702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:21:37.130226 6702 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:21:37.132238 6702 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:21:37.141129 6702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53: Generated new TSK 0
I20260925 01:21:37.142227 6702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:21:37.208066 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:34647
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:46617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:37.546830 6714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:37.547166 6714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:37.547330 6714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:37.555203 6714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:37.555375 6714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:37.555554 6714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:21:37.565416 6714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:46617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:34647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:37.566819 6714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:37.568171 6714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:37.582114 6722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:37.582137 6720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:37.581898 6719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:37.583137 6714 server_base.cc:1034] running on GCE node
I20260925 01:21:37.583695 6714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:37.584571 6714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:37.585816 6714 hybrid_clock.cc:648] HybridClock initialized: now 1790299297585770 us; error 32 us; skew 500 ppm
I20260925 01:21:37.586154 6714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:37 dist-test-slave-s12s krb5kdc[6608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299297, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:37.589357 6714 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:21:37.590919 6714 webserver.cc:460] Webserver started at http://127.20.102.129:45381/ using document root <none> and password file <none>
I20260925 01:21:37.591629 6714 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:37.591784 6714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:37.592119 6714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:37.595376 6714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "f9121f0bd61443aea5decf9d4cf4c07b"
format_stamp: "Formatted at 2026-09-25 01:21:37 on dist-test-slave-s12s"
server_key: "2b97001d583d5df58f4e0daba06a9e6c"
server_key_iv: "9bc1793e23bbfa0ae26529e1b8486581"
server_key_version: "encryptionkey@0"
I20260925 01:21:37.596392 6714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "f9121f0bd61443aea5decf9d4cf4c07b"
format_stamp: "Formatted at 2026-09-25 01:21:37 on dist-test-slave-s12s"
server_key: "2b97001d583d5df58f4e0daba06a9e6c"
server_key_iv: "9bc1793e23bbfa0ae26529e1b8486581"
server_key_version: "encryptionkey@0"
I20260925 01:21:37.601912 6714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 01:21:37.605507 6729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:37.606891 6714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:21:37.607075 6714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "f9121f0bd61443aea5decf9d4cf4c07b"
format_stamp: "Formatted at 2026-09-25 01:21:37 on dist-test-slave-s12s"
server_key: "2b97001d583d5df58f4e0daba06a9e6c"
server_key_iv: "9bc1793e23bbfa0ae26529e1b8486581"
server_key_version: "encryptionkey@0"
I20260925 01:21:37.607297 6714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:37.631925 6714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:37.636144 6714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:37.636441 6714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:37.637534 6714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:37.638924 6714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:37.638998 6714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:37.639133 6714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:37.639202 6714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:37.673476 6714 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:44485
I20260925 01:21:37.673565 6841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:44485 every 8 connection(s)
I20260925 01:21:37.674809 6714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:21:37.676450 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 6714
I20260925 01:21:37.676647 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:21:37.677166 20890 external_mini_cluster.cc:1428] Setting key 01bd2a37721777dfa56427818a40b446
Sep 25 01:21:37 dist-test-slave-s12s krb5kdc[6608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299297, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:21:37.696868 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:37.677858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:36407 (local address 127.20.102.190:34647)
0925 01:21:37.678316 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:37.678323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:37.679585 (+ 1262us) server_negotiation.cc:407] Connection header received
0925 01:21:37.681619 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:37.681624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:37.681720 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:37.681844 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:37.683462 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:37.684202 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:37.685864 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:37.686277 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:37.686416 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:37.690674 (+ 4258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:37.690679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:37.690682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:37.690717 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:37.693449 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:37.694184 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:37.694195 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:37.694197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:37.694321 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:37.694772 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:37.694781 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:37.694783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:37.695041 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:37.695204 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:37.696276 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0925 01:21:37.696521 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":204,"threads_started":1}
I20260925 01:21:37.698444 6845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:37.678549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34647 (local address 127.20.102.129:36407)
0925 01:21:37.679402 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:37.679453 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:37.681336 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:37.682093 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:37.682109 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:37.682654 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:37.683254 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:37.683272 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:37.684377 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:37.684382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:37.685698 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:37.685716 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:37.686527 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:37.686535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:37.686859 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:37.687670 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:21:37.687707 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:37.690443 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:37.693582 (+ 3139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:37.693591 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:37.693617 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:37.694041 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:37.694435 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:37.694440 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:37.694444 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:37.694645 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:37.695221 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:37.695253 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:37.695778 (+ 525us) client_negotiation.cc:769] Sending connection context
0925 01:21:37.696134 (+ 356us) client_negotiation.cc:240] Negotiation successful
0925 01:21:37.696810 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":345,"threads_started":1}
I20260925 01:21:37.700150 6842 heartbeater.cc:347] Connected to a master server at 127.20.102.190:34647
I20260925 01:21:37.700510 6842 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:37.701586 6842 heartbeater.cc:511] Master 127.20.102.190:34647 requested a full tablet report, sending...
I20260925 01:21:37.704073 6653 ts_manager.cc:194] Registered new tserver with Master: f9121f0bd61443aea5decf9d4cf4c07b (127.20.102.129:44485)
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
I20260925 01:21:37.705709 6653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:36407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:21:37.736821 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:34647
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:46617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:38.079296 6848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:38.079594 6848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:38.079692 6848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:38.087194 6848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:38.087319 6848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:38.087437 6848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:21:38.098011 6848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:46617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:34647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:38.099316 6848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:38.100586 6848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:38.114162 6853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:38.114114 6854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:38.114207 6856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:38.115520 6848 server_base.cc:1034] running on GCE node
I20260925 01:21:38.116171 6848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:38.117233 6848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:38.118460 6848 hybrid_clock.cc:648] HybridClock initialized: now 1790299298118390 us; error 62 us; skew 500 ppm
I20260925 01:21:38.118788 6848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:38 dist-test-slave-s12s krb5kdc[6608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299298, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:38.122151 6848 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:21:38.123828 6848 webserver.cc:460] Webserver started at http://127.20.102.130:41627/ using document root <none> and password file <none>
I20260925 01:21:38.124552 6848 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:38.124745 6848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:38.125077 6848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:38.128026 6848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "f3691139230a4ffaa9f874572d19d65d"
format_stamp: "Formatted at 2026-09-25 01:21:38 on dist-test-slave-s12s"
server_key: "2ce1f442c4f620f4090d228f5a6f90f0"
server_key_iv: "01eceae2c2b5878f7248799da5547b9b"
server_key_version: "encryptionkey@0"
I20260925 01:21:38.128940 6848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "f3691139230a4ffaa9f874572d19d65d"
format_stamp: "Formatted at 2026-09-25 01:21:38 on dist-test-slave-s12s"
server_key: "2ce1f442c4f620f4090d228f5a6f90f0"
server_key_iv: "01eceae2c2b5878f7248799da5547b9b"
server_key_version: "encryptionkey@0"
I20260925 01:21:38.134217 6848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 01:21:38.137854 6863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:38.139256 6848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:21:38.139438 6848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "f3691139230a4ffaa9f874572d19d65d"
format_stamp: "Formatted at 2026-09-25 01:21:38 on dist-test-slave-s12s"
server_key: "2ce1f442c4f620f4090d228f5a6f90f0"
server_key_iv: "01eceae2c2b5878f7248799da5547b9b"
server_key_version: "encryptionkey@0"
I20260925 01:21:38.139653 6848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:38.156309 6848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:38.160336 6848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:38.160629 6848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:38.161687 6848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:38.163058 6848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:38.163170 6848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:38.163324 6848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:38.163396 6848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:38.197624 6848 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:46537
I20260925 01:21:38.197645 6975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:46537 every 8 connection(s)
I20260925 01:21:38.198974 6848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:21:38.202713 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 6848
I20260925 01:21:38.202905 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:21:38.203466 20890 external_mini_cluster.cc:1428] Setting key 06cbde68eedc0ade232708a57045bada
Sep 25 01:21:38 dist-test-slave-s12s krb5kdc[6608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299298, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:21:38.220099 6978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.201627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:39275 (local address 127.20.102.190:34647)
0925 01:21:38.202089 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:38.202094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:38.203090 (+ 996us) server_negotiation.cc:407] Connection header received
0925 01:21:38.205665 (+ 2575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:38.205671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:38.205763 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:38.205890 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:38.207386 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.208123 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.209733 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.210175 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.210312 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.214149 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:38.214154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:38.214157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:38.214190 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:38.216827 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:38.217565 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:38.217571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:38.217580 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:38.217775 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:38.218223 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:38.218228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:38.218231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:38.218573 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:38.218728 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:38.219474 (+ 746us) server_negotiation.cc:299] Negotiation successful
0925 01:21:38.219705 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":244,"threads_started":1}
I20260925 01:21:38.221282 6979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.202011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34647 (local address 127.20.102.130:39275)
0925 01:21:38.202876 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:38.202940 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:38.205393 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:38.206127 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:38.206135 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:38.206651 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:38.207221 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:38.207237 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.208294 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.208300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:38.209589 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:38.209601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.210414 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.210421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:38.210655 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.211402 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:21:38.211426 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:38.213903 (+ 2477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:38.216963 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:38.216975 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:38.216992 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:38.217429 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:38.217888 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:38.217893 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:38.217897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:38.218098 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:38.218706 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:38.218715 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:38.219099 (+ 384us) client_negotiation.cc:769] Sending connection context
0925 01:21:38.219385 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 01:21:38.220015 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":329,"threads_started":1}
I20260925 01:21:38.222882 6976 heartbeater.cc:347] Connected to a master server at 127.20.102.190:34647
I20260925 01:21:38.223205 6976 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:38.224031 6976 heartbeater.cc:511] Master 127.20.102.190:34647 requested a full tablet report, sending...
I20260925 01:21:38.225648 6653 ts_manager.cc:194] Registered new tserver with Master: f3691139230a4ffaa9f874572d19d65d (127.20.102.130:46537)
I20260925 01:21:38.226927 6653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:39275
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:21:38.262216 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:34647
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:46617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:38.593137 6982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:38.593422 6982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:38.593585 6982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:38.601382 6982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:38.601560 6982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:38.601760 6982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:21:38.611869 6982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:46617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:34647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:38.613356 6982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:38.614738 6982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:38.628959 6987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:38.628916 6988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:38.628930 6990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:38.629870 6982 server_base.cc:1034] running on GCE node
I20260925 01:21:38.630473 6982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:38.631428 6982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:38.632658 6982 hybrid_clock.cc:648] HybridClock initialized: now 1790299298632605 us; error 40 us; skew 500 ppm
I20260925 01:21:38.633034 6982 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:38 dist-test-slave-s12s krb5kdc[6608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299298, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:38.636206 6982 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:21:38.637845 6982 webserver.cc:460] Webserver started at http://127.20.102.131:40887/ using document root <none> and password file <none>
I20260925 01:21:38.638530 6982 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:38.638676 6982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:38.638995 6982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:38.641960 6982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93"
format_stamp: "Formatted at 2026-09-25 01:21:38 on dist-test-slave-s12s"
server_key: "e3ad791480431b7fddcbfd402f3edc5d"
server_key_iv: "2e0dc92362712f9c0b9fa7f6fb36177c"
server_key_version: "encryptionkey@0"
I20260925 01:21:38.642719 6982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93"
format_stamp: "Formatted at 2026-09-25 01:21:38 on dist-test-slave-s12s"
server_key: "e3ad791480431b7fddcbfd402f3edc5d"
server_key_iv: "2e0dc92362712f9c0b9fa7f6fb36177c"
server_key_version: "encryptionkey@0"
I20260925 01:21:38.647819 6982 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:21:38.651248 6997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:38.652588 6982 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:21:38.652799 6982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93"
format_stamp: "Formatted at 2026-09-25 01:21:38 on dist-test-slave-s12s"
server_key: "e3ad791480431b7fddcbfd402f3edc5d"
server_key_iv: "2e0dc92362712f9c0b9fa7f6fb36177c"
server_key_version: "encryptionkey@0"
I20260925 01:21:38.653007 6982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:38.677182 6982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:38.681564 6982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:38.681865 6982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:38.682870 6982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:38.684183 6982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:38.684286 6982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:38.684441 6982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:38.684514 6982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:38.709549 6842 heartbeater.cc:503] Master 127.20.102.190:34647 was elected leader, sending a full tablet report...
I20260925 01:21:38.718344 6982 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:38887
I20260925 01:21:38.718386 7109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:38887 every 8 connection(s)
I20260925 01:21:38.719794 6982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:21:38.727816 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 6982
I20260925 01:21:38.728012 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:21:38.728462 20890 external_mini_cluster.cc:1428] Setting key c987533eaa693155f7e1d76a0514f677
Sep 25 01:21:38 dist-test-slave-s12s krb5kdc[6608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299298, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:21:38.740098 7112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.722433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:51369 (local address 127.20.102.190:34647)
0925 01:21:38.722878 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:38.722882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:38.723762 (+ 880us) server_negotiation.cc:407] Connection header received
0925 01:21:38.725670 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:38.725676 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:38.725780 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:38.725900 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:38.727362 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.728290 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.730277 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.730708 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.730838 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.734244 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:38.734255 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:38.734258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:38.734290 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:38.737053 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:38.737821 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:38.737825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:38.737828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:38.737949 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:38.738388 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:38.738393 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:38.738395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:38.738656 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:38.738818 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:38.739501 (+ 683us) server_negotiation.cc:299] Negotiation successful
0925 01:21:38.739721 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":245,"threads_started":1}
I20260925 01:21:38.741107 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.722893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:34647 (local address 127.20.102.131:51369)
0925 01:21:38.723575 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:38.723612 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:38.725403 (+ 1791us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:38.726131 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:38.726139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:38.726650 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:38.727188 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:38.727205 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.728445 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.728449 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:38.730141 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:38.730153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.730935 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.730941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:38.731176 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.731847 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:21:38.731870 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:38.734017 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:38.737202 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:38.737211 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:38.737227 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:38.737684 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:38.738061 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:38.738066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:38.738070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:38.738261 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:38.738800 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:38.738808 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:38.739138 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 01:21:38.739391 (+ 253us) client_negotiation.cc:240] Negotiation successful
0925 01:21:38.739922 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":256,"threads_started":1}
I20260925 01:21:38.742935 7110 heartbeater.cc:347] Connected to a master server at 127.20.102.190:34647
I20260925 01:21:38.743283 7110 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:38.744459 7110 heartbeater.cc:511] Master 127.20.102.190:34647 requested a full tablet report, sending...
I20260925 01:21:38.746133 6652 ts_manager.cc:194] Registered new tserver with Master: 6191fccd1fdd4e379cc6dd1f36d0fa93 (127.20.102.131:38887)
I20260925 01:21:38.746997 6652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:51369
I20260925 01:21:38.756389 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:21:38.773710 7112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.760601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53242 (local address 127.20.102.190:34647)
0925 01:21:38.760812 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:38.760816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:38.761203 (+ 387us) server_negotiation.cc:407] Connection header received
0925 01:21:38.761391 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:38.761396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:38.761459 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:38.761553 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:38.762989 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.763739 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.765103 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.765504 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.765620 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.768188 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:38.768193 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:38.768197 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:38.768231 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:38.770805 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:38.771660 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:38.771664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:38.771666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:38.771784 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:38.772176 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:38.772179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:38.772181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:38.772500 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:38.772795 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:38.773297 (+ 502us) server_negotiation.cc:299] Negotiation successful
0925 01:21:38.773452 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:21:38.777673 6652 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:21:38.784919 6652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:21:38.810083 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.797736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:46537 (local address 127.0.0.1:56080)
0925 01:21:38.799062 (+ 1326us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:38.799099 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:38.799241 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:38.800008 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:38.800021 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:38.800065 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:38.800708 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:38.800728 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.803243 (+ 2515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.803252 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:38.807032 (+ 3780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:38.807053 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.809059 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.809066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:38.809281 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.809338 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 01:21:38.809446 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 01:21:38.809654 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1144,"thread_start_us":1016,"threads_started":1}
I20260925 01:21:38.811167 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.797703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.20.102.130:46537)
0925 01:21:38.798559 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:38.798573 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:38.799207 (+ 634us) server_negotiation.cc:407] Connection header received
0925 01:21:38.799436 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:38.799445 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:38.799741 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:38.799985 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:38.800989 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.803054 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.807249 (+ 4195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.808831 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.810365 (+ 1534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.810573 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:38.810661 (+ 88us) server_negotiation.cc:299] Negotiation successful
0925 01:21:38.810868 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":275,"threads_started":1}
I20260925 01:21:38.813921 7125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.799492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:44485 (local address 127.0.0.1:51518)
0925 01:21:38.805091 (+ 5599us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:38.805110 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:38.805260 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:38.805917 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:38.805921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:38.805946 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:38.806349 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:38.806359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.807956 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.807965 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:38.810417 (+ 2452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:38.810433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.813052 (+ 2619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.813059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:38.813235 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.813272 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 01:21:38.813411 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 01:21:38.813545 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5468,"thread_start_us":5331,"threads_started":1}
I20260925 01:21:38.813946 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.799865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51518 (local address 127.20.102.129:44485)
0925 01:21:38.803748 (+ 3883us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:38.803755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:38.805225 (+ 1470us) server_negotiation.cc:407] Connection header received
0925 01:21:38.805433 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:38.805439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:38.805729 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:38.805934 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:38.806506 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.807795 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.810599 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.812835 (+ 2236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.813051 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.813249 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:38.813456 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 01:21:38.813615 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3707,"thread_start_us":3530,"threads_started":1}
I20260925 01:21:38.815395 6911 tablet_service.cc:1467] Processing CreateTablet for tablet c66c368e502e4b0c94cef7d79d417b58 (DEFAULT_TABLE table=test-table [id=b21d457de98c4a669a42016013c808e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:38.817030 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.807706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:38887 (local address 127.0.0.1:57186)
0925 01:21:38.808353 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:38.808368 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:38.808508 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:38.809430 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:38.809434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:38.809457 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:38.809980 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:38.809999 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.811687 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.811692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:38.813794 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:38.813808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.816150 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.816156 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:38.816327 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.816360 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 01:21:38.816459 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 01:21:38.816598 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":34,"thread_start_us":341,"threads_started":1}
I20260925 01:21:38.817107 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.805499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57186 (local address 127.20.102.131:38887)
0925 01:21:38.807705 (+ 2206us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:38.807713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:38.808471 (+ 758us) server_negotiation.cc:407] Connection header received
0925 01:21:38.808726 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:38.808734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:38.808968 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:38.809158 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:38.810200 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.811532 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.813983 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.815329 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.815509 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.815687 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:38.816539 (+ 852us) server_negotiation.cc:299] Negotiation successful
0925 01:21:38.816781 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2051,"thread_start_us":1894,"threads_started":1}
I20260925 01:21:38.818193 6911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66c368e502e4b0c94cef7d79d417b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:38.819252 6777 tablet_service.cc:1467] Processing CreateTablet for tablet c66c368e502e4b0c94cef7d79d417b58 (DEFAULT_TABLE table=test-table [id=b21d457de98c4a669a42016013c808e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:38.820519 6777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66c368e502e4b0c94cef7d79d417b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:38.821053 7045 tablet_service.cc:1467] Processing CreateTablet for tablet c66c368e502e4b0c94cef7d79d417b58 (DEFAULT_TABLE table=test-table [id=b21d457de98c4a669a42016013c808e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:38.822405 7045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c66c368e502e4b0c94cef7d79d417b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:38.832077 7129 tablet_bootstrap.cc:492] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b: Bootstrap starting.
I20260925 01:21:38.835261 7129 tablet_bootstrap.cc:654] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:38.836150 7130 tablet_bootstrap.cc:492] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d: Bootstrap starting.
I20260925 01:21:38.836650 7129 log.cc:824] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:38.837659 7131 tablet_bootstrap.cc:492] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93: Bootstrap starting.
I20260925 01:21:38.840051 7130 tablet_bootstrap.cc:654] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:38.841382 7129 tablet_bootstrap.cc:492] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b: No bootstrap required, opened a new log
I20260925 01:21:38.841394 7131 tablet_bootstrap.cc:654] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:38.841436 7130 log.cc:824] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:38.841691 7129 ts_tablet_manager.cc:1397] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260925 01:21:38.842672 7131 log.cc:824] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:38.844667 7130 tablet_bootstrap.cc:492] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d: No bootstrap required, opened a new log
I20260925 01:21:38.844993 7130 ts_tablet_manager.cc:1397] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 01:21:38.845430 7131 tablet_bootstrap.cc:492] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93: No bootstrap required, opened a new log
I20260925 01:21:38.845677 7131 ts_tablet_manager.cc:1397] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 01:21:38.852696 7129 raft_consensus.cc:348] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3691139230a4ffaa9f874572d19d65d" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 46537 } } peers { permanent_uuid: "f9121f0bd61443aea5decf9d4cf4c07b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44485 } } peers { permanent_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38887 } }
I20260925 01:21:38.853211 7129 raft_consensus.cc:374] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:38.853370 7129 raft_consensus.cc:729] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9121f0bd61443aea5decf9d4cf4c07b, State: Initialized, Role: FOLLOWER
I20260925 01:21:38.853438 7131 raft_consensus.cc:348] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3691139230a4ffaa9f874572d19d65d" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 46537 } } peers { permanent_uuid: "f9121f0bd61443aea5decf9d4cf4c07b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44485 } } peers { permanent_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38887 } }
I20260925 01:21:38.853899 7131 raft_consensus.cc:374] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:38.854025 7131 raft_consensus.cc:729] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6191fccd1fdd4e379cc6dd1f36d0fa93, State: Initialized, Role: FOLLOWER
I20260925 01:21:38.854084 7129 consensus_queue.cc:260] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3691139230a4ffaa9f874572d19d65d" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 46537 } } peers { permanent_uuid: "f9121f0bd61443aea5decf9d4cf4c07b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44485 } } peers { permanent_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38887 } }
I20260925 01:21:38.854652 7131 consensus_queue.cc:260] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3691139230a4ffaa9f874572d19d65d" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 46537 } } peers { permanent_uuid: "f9121f0bd61443aea5decf9d4cf4c07b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44485 } } peers { permanent_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38887 } }
I20260925 01:21:38.855605 7130 raft_consensus.cc:348] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3691139230a4ffaa9f874572d19d65d" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 46537 } } peers { permanent_uuid: "f9121f0bd61443aea5decf9d4cf4c07b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44485 } } peers { permanent_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38887 } }
I20260925 01:21:38.856076 7130 raft_consensus.cc:374] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:38.856190 7129 ts_tablet_manager.cc:1428] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260925 01:21:38.856192 7130 raft_consensus.cc:729] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3691139230a4ffaa9f874572d19d65d, State: Initialized, Role: FOLLOWER
I20260925 01:21:38.856824 7130 consensus_queue.cc:260] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3691139230a4ffaa9f874572d19d65d" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 46537 } } peers { permanent_uuid: "f9121f0bd61443aea5decf9d4cf4c07b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44485 } } peers { permanent_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38887 } }
I20260925 01:21:38.857524 7110 heartbeater.cc:503] Master 127.20.102.190:34647 was elected leader, sending a full tablet report...
I20260925 01:21:38.859592 6976 heartbeater.cc:503] Master 127.20.102.190:34647 was elected leader, sending a full tablet report...
I20260925 01:21:38.860344 7130 ts_tablet_manager.cc:1428] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260925 01:21:38.861534 7131 ts_tablet_manager.cc:1428] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93: Time spent starting tablet: real 0.016s user 0.010s sys 0.000s
I20260925 01:21:38.919364 7136 raft_consensus.cc:482] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:21:38.919696 7136 raft_consensus.cc:504] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3691139230a4ffaa9f874572d19d65d" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 46537 } } peers { permanent_uuid: "f9121f0bd61443aea5decf9d4cf4c07b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44485 } } peers { permanent_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38887 } }
I20260925 01:21:38.921458 7136 leader_election.cc:288] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3691139230a4ffaa9f874572d19d65d (127.20.102.130:46537), f9121f0bd61443aea5decf9d4cf4c07b (127.20.102.129:44485)
W20260925 01:21:38.927356 6843 tablet.cc:2369] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:21:38.928563 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.921921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:40893 (local address 127.20.102.130:46537)
0925 01:21:38.922186 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:38.922192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:38.922216 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:21:38.922295 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:38.922301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:38.922374 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:38.922481 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:38.923112 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.924235 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.926444 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.927604 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.927759 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.927855 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:38.928168 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 01:21:38.928293 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:21:38.928802 7113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.921769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:46537 (local address 127.20.102.131:40893)
0925 01:21:38.921987 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:38.922010 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:38.922132 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:38.922682 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:38.922686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:38.922712 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:38.922871 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:38.922902 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.924376 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.924382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:38.926279 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:38.926291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.927789 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.927795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:38.927998 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.928029 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 01:21:38.928140 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 01:21:38.928268 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":49}
I20260925 01:21:38.928864 7138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.922147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:44485 (local address 127.20.102.131:41327)
0925 01:21:38.922824 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:38.922839 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:38.922943 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:38.923293 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:38.923298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:38.923323 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:38.923709 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:38.923721 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.924899 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.924904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:38.926592 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:38.926606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.928024 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.928030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:38.928224 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.928255 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 01:21:38.928349 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 01:21:38.928475 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":384,"threads_started":1}
I20260925 01:21:38.929455 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.922340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:41327 (local address 127.20.102.129:44485)
0925 01:21:38.922597 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:38.922604 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:38.922933 (+ 329us) server_negotiation.cc:407] Connection header received
0925 01:21:38.923117 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:38.923120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:38.923177 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:38.923245 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:38.923862 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.924735 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.926747 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.927672 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.927833 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.927907 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:38.928383 (+ 476us) server_negotiation.cc:299] Negotiation successful
0925 01:21:38.929133 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:21:38.930305 6797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66c368e502e4b0c94cef7d79d417b58" candidate_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9121f0bd61443aea5decf9d4cf4c07b" is_pre_election: true
I20260925 01:21:38.930823 6797 raft_consensus.cc:2454] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6191fccd1fdd4e379cc6dd1f36d0fa93 in term 0.
I20260925 01:21:38.931298 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66c368e502e4b0c94cef7d79d417b58" candidate_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3691139230a4ffaa9f874572d19d65d" is_pre_election: true
I20260925 01:21:38.931604 7000 leader_election.cc:302] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6191fccd1fdd4e379cc6dd1f36d0fa93, f9121f0bd61443aea5decf9d4cf4c07b; no voters:
I20260925 01:21:38.931936 6931 raft_consensus.cc:2454] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6191fccd1fdd4e379cc6dd1f36d0fa93 in term 0.
I20260925 01:21:38.932024 7136 raft_consensus.cc:2781] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:21:38.932185 7136 raft_consensus.cc:482] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:21:38.932281 7136 raft_consensus.cc:3037] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:38.934605 7136 raft_consensus.cc:504] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3691139230a4ffaa9f874572d19d65d" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 46537 } } peers { permanent_uuid: "f9121f0bd61443aea5decf9d4cf4c07b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44485 } } peers { permanent_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38887 } }
I20260925 01:21:38.935313 7136 leader_election.cc:288] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [CANDIDATE]: Term 1 election: Requested vote from peers f3691139230a4ffaa9f874572d19d65d (127.20.102.130:46537), f9121f0bd61443aea5decf9d4cf4c07b (127.20.102.129:44485)
I20260925 01:21:38.935775 6931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66c368e502e4b0c94cef7d79d417b58" candidate_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3691139230a4ffaa9f874572d19d65d"
I20260925 01:21:38.935849 6797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c66c368e502e4b0c94cef7d79d417b58" candidate_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9121f0bd61443aea5decf9d4cf4c07b"
I20260925 01:21:38.935956 6931 raft_consensus.cc:3037] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:38.936120 6797 raft_consensus.cc:3037] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:38.938419 6931 raft_consensus.cc:2454] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6191fccd1fdd4e379cc6dd1f36d0fa93 in term 1.
I20260925 01:21:38.938563 6797 raft_consensus.cc:2454] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6191fccd1fdd4e379cc6dd1f36d0fa93 in term 1.
I20260925 01:21:38.938836 7000 leader_election.cc:302] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6191fccd1fdd4e379cc6dd1f36d0fa93, f3691139230a4ffaa9f874572d19d65d; no voters:
I20260925 01:21:38.939141 7136 raft_consensus.cc:2781] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:38.939652 7136 raft_consensus.cc:686] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [term 1 LEADER]: Becoming Leader. State: Replica: 6191fccd1fdd4e379cc6dd1f36d0fa93, State: Running, Role: LEADER
I20260925 01:21:38.940142 7136 consensus_queue.cc:237] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3691139230a4ffaa9f874572d19d65d" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 46537 } } peers { permanent_uuid: "f9121f0bd61443aea5decf9d4cf4c07b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44485 } } peers { permanent_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38887 } }
I20260925 01:21:38.945510 6653 catalog_manager.cc:5495] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6191fccd1fdd4e379cc6dd1f36d0fa93 (127.20.102.131). New cstate: current_term: 1 leader_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3691139230a4ffaa9f874572d19d65d" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 46537 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9121f0bd61443aea5decf9d4cf4c07b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6191fccd1fdd4e379cc6dd1f36d0fa93" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 38887 } health_report { overall_health: HEALTHY } } }
W20260925 01:21:38.950989 6977 tablet.cc:2369] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:21:38.967440 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:38.962055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.20.102.131:38887)
0925 01:21:38.962265 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:38.962271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:38.962290 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:21:38.962372 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:38.962375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:38.962457 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:38.962619 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:21:38.962999 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.963752 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.965220 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:38.965627 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:38.965760 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:38.966319 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:21:38.966813 (+ 494us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:21:38.967010 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:38.967168 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 01:21:38.967221 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 01:21:38.971379 7111 tablet.cc:2369] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:21:38.975687 6931 raft_consensus.cc:1261] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d [term 1 FOLLOWER]: Refusing update from remote peer 6191fccd1fdd4e379cc6dd1f36d0fa93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:21:38.975680 6797 raft_consensus.cc:1261] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b [term 1 FOLLOWER]: Refusing update from remote peer 6191fccd1fdd4e379cc6dd1f36d0fa93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:21:38.976634 7139 consensus_queue.cc:1035] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9121f0bd61443aea5decf9d4cf4c07b" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 44485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:21:38.977074 7136 consensus_queue.cc:1035] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3691139230a4ffaa9f874572d19d65d" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 46537 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:21:39.002342 6652 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:21:39.002660 6652 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:21:39.006299 6652 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e91024eabd9843eda595e78c733a9f53: Sending DeleteTablet for 3 replicas of tablet c66c368e502e4b0c94cef7d79d417b58
I20260925 01:21:39.007483 6911 tablet_service.cc:1514] Processing DeleteTablet for tablet c66c368e502e4b0c94cef7d79d417b58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:38 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:56080
I20260925 01:21:39.007702 6777 tablet_service.cc:1514] Processing DeleteTablet for tablet c66c368e502e4b0c94cef7d79d417b58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:38 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:51518
I20260925 01:21:39.008368 7045 tablet_service.cc:1514] Processing DeleteTablet for tablet c66c368e502e4b0c94cef7d79d417b58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:38 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:57186
I20260925 01:21:39.010852 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 6714
Sep 25 01:21:39 dist-test-slave-s12s krb5kdc[6608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:21:39.038727 7154 tablet_replica.cc:330] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93: stopping tablet replica
I20260925 01:21:39.039451 7154 raft_consensus.cc:2229] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:21:39.040318 7154 raft_consensus.cc:2258] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:39.041369 7152 tablet_replica.cc:330] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d: stopping tablet replica
I20260925 01:21:39.042161 7152 raft_consensus.cc:2229] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:21:39.043711 7152 raft_consensus.cc:2258] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:39.045338 7126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:39.012508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.20.102.129:44485)
0925 01:21:39.012758 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:39.012767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:39.023892 (+ 11125us) server_negotiation.cc:407] Connection header received
0925 01:21:39.024403 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:39.024412 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:39.024526 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:39.024624 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:39.026549 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:39.027763 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:39.029634 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:39.030424 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:39.030525 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:39.037102 (+ 6577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:39.037126 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:39.037153 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:39.037206 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:39.041614 (+ 4408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:39.042499 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:39.042508 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:39.042511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:39.042712 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:39.043219 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:39.043225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:39.043228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:39.043753 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:39.044055 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:39.044671 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 01:21:39.044942 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:21:39.045374 7154 ts_tablet_manager.cc:1905] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:39.046844 6837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:39.051123 7152 ts_tablet_manager.cc:1905] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:39.073159 7152 ts_tablet_manager.cc:1918] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:39.073392 7152 log.cc:1196] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/c66c368e502e4b0c94cef7d79d417b58
I20260925 01:21:39.074080 7152 ts_tablet_manager.cc:1939] T c66c368e502e4b0c94cef7d79d417b58 P f3691139230a4ffaa9f874572d19d65d: Deleting consensus metadata
I20260925 01:21:39.076227 7154 ts_tablet_manager.cc:1918] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:39.076566 7154 log.cc:1196] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/c66c368e502e4b0c94cef7d79d417b58
I20260925 01:21:39.076700 6639 catalog_manager.cc:4841] TS f3691139230a4ffaa9f874572d19d65d (127.20.102.130:46537): tablet c66c368e502e4b0c94cef7d79d417b58 (table test-table [id=b21d457de98c4a669a42016013c808e3]) successfully deleted
I20260925 01:21:39.079761 7154 ts_tablet_manager.cc:1939] T c66c368e502e4b0c94cef7d79d417b58 P 6191fccd1fdd4e379cc6dd1f36d0fa93: Deleting consensus metadata
I20260925 01:21:39.081678 6639 catalog_manager.cc:4841] TS 6191fccd1fdd4e379cc6dd1f36d0fa93 (127.20.102.131:38887): tablet c66c368e502e4b0c94cef7d79d417b58 (table test-table [id=b21d457de98c4a669a42016013c808e3]) successfully deleted
I20260925 01:21:39.169664 7153 tablet_replica.cc:330] T c66c368e502e4b0c94cef7d79d417b58 P f9121f0bd61443aea5decf9d4cf4c07b: stopping tablet replica
I20260925 01:21:39.170035 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 6714
W20260925 01:21:39.188294 6639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.102.129:44485 (error 108)
I20260925 01:21:39.189080 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 6848
I20260925 01:21:39.190591 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:39.190079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:44485 (local address 127.0.0.1:51544)
0925 01:21:39.190277 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:39.190402 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:44485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 01:21:39.190871 6639 catalog_manager.cc:4584] TS f9121f0bd61443aea5decf9d4cf4c07b (127.20.102.129:44485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c66c368e502e4b0c94cef7d79d417b58: Network error: Client connection negotiation failed: client connection to 127.20.102.129:44485: connect: Connection refused (error 111)
Sep 25 01:21:39 dist-test-slave-s12s krb5kdc[6608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:21:39.208580 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:39.189974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56090 (local address 127.20.102.130:46537)
0925 01:21:39.190398 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:39.190405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:39.190427 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:21:39.190500 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:39.190507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:39.190600 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:39.190706 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:39.192307 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:39.193577 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:39.195109 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:39.195758 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:39.195890 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:39.201493 (+ 5603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:39.201508 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:39.201532 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:39.201581 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:39.205489 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:39.206325 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:39.206334 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:39.206338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:39.206482 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:39.206889 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:39.206894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:39.206897 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:39.207352 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:39.207624 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:39.207977 (+ 353us) server_negotiation.cc:299] Negotiation successful
0925 01:21:39.208223 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285}
I20260925 01:21:39.209566 6971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:39.244637 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:39.244146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:44485 (local address 127.0.0.1:51546)
0925 01:21:39.244370 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:39.244473 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:44485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 01:21:39.245798 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:39.245439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:44485 (local address 127.0.0.1:51550)
0925 01:21:39.245598 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:39.245689 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:44485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 01:21:39.306335 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:39.305841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:44485 (local address 127.0.0.1:51562)
0925 01:21:39.306070 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:39.306171 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:44485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 01:21:39.307449 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:39.307096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:44485 (local address 127.0.0.1:51572)
0925 01:21:39.307252 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:39.307330 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:44485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 01:21:39.359937 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 6848
I20260925 01:21:39.375399 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 6982
Sep 25 01:21:39 dist-test-slave-s12s krb5kdc[6608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:21:39.388242 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:39.387780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:44485 (local address 127.0.0.1:51576)
0925 01:21:39.388010 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:39.388090 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:44485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 01:21:39.389582 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:39.389101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:44485 (local address 127.0.0.1:51588)
0925 01:21:39.389347 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:39.389460 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:44485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260925 01:21:39.394942 7127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:39.376270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57200 (local address 127.20.102.131:38887)
0925 01:21:39.376483 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:39.376491 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:39.376519 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:21:39.376617 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:39.376623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:39.376739 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:39.376845 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:39.378310 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:39.379517 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:39.380979 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:39.381694 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:39.381843 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:39.386783 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:39.386797 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:39.386814 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:39.386860 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:39.391097 (+ 4237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:39.391957 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:39.391964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:39.391967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:39.392156 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:39.392661 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:39.392668 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:39.392730 (+ 62us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:39.393403 (+ 673us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:39.393909 (+ 506us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:39.394217 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 01:21:39.394508 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:21:39.395841 7105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:39.520840 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 6982
I20260925 01:21:39.541903 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 6621
I20260925 01:21:39.543094 6683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:39.654665 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:39.654103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:44485 (local address 127.0.0.1:51602)
0925 01:21:39.654318 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:39.654400 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:44485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 01:21:39.654834 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 6621
2026-09-25T01:21:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4526 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T01:21:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:21:39Z Disabled control of system clock
I20260925 01:21:39.734350 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:44783
--webserver_interface=127.20.102.190
--webserver_port=0
--builtin_ntp_servers=127.20.102.148:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:44783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:21:40.067317 7164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:40.067616 7164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:40.067699 7164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:40.074955 7164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:21:40.075070 7164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:40.075109 7164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:40.075172 7164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:21:40.075217 7164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:21:40.084343 7164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:44783
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:44783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:40.085659 7164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:40.087417 7164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:40.100178 7172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:40.100243 7169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:40.100266 7170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:40.101123 7164 server_base.cc:1034] running on GCE node
I20260925 01:21:40.101900 7164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:40.103300 7164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:40.104537 7164 hybrid_clock.cc:648] HybridClock initialized: now 1790299300104479 us; error 55 us; skew 500 ppm
I20260925 01:21:40.104933 7164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:21:40.107297 7164 webserver.cc:460] Webserver started at http://127.20.102.190:40037/ using document root <none> and password file <none>
I20260925 01:21:40.107987 7164 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:40.108119 7164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:40.108455 7164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:40.111410 7164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "b51d17a9f34e44f38d65cad61e17782c"
format_stamp: "Formatted at 2026-09-25 01:21:40 on dist-test-slave-s12s"
server_key: "a4867291f075b68dfbdbe8194055f19e"
server_key_iv: "93090e00f540535c7c2c93bc8cc3b3ba"
server_key_version: "encryptionkey@0"
I20260925 01:21:40.112223 7164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "b51d17a9f34e44f38d65cad61e17782c"
format_stamp: "Formatted at 2026-09-25 01:21:40 on dist-test-slave-s12s"
server_key: "a4867291f075b68dfbdbe8194055f19e"
server_key_iv: "93090e00f540535c7c2c93bc8cc3b3ba"
server_key_version: "encryptionkey@0"
I20260925 01:21:40.117498 7164 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260925 01:21:40.120955 7178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:40.122269 7164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 01:21:40.122475 7164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "b51d17a9f34e44f38d65cad61e17782c"
format_stamp: "Formatted at 2026-09-25 01:21:40 on dist-test-slave-s12s"
server_key: "a4867291f075b68dfbdbe8194055f19e"
server_key_iv: "93090e00f540535c7c2c93bc8cc3b3ba"
server_key_version: "encryptionkey@0"
I20260925 01:21:40.122709 7164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:40.146478 7164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:40.147390 7164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:40.147703 7164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:40.167150 7164 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:44783
I20260925 01:21:40.167179 7229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:44783 every 8 connection(s)
I20260925 01:21:40.168578 7164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:21:40.172498 7230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:40.178263 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 7164
I20260925 01:21:40.178447 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:21:40.178975 20890 external_mini_cluster.cc:1428] Setting key 8eac58bbda5f9ca7d1f1c2336a7fdbb4
I20260925 01:21:40.184051 7230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c: Bootstrap starting.
I20260925 01:21:40.187271 7230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:40.188299 7230 log.cc:824] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:40.191738 7230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c: No bootstrap required, opened a new log
I20260925 01:21:40.193421 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:40.181100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.20.102.190:44783)
0925 01:21:40.182125 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:40.182135 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:40.182180 (+ 45us) server_negotiation.cc:407] Connection header received
0925 01:21:40.183828 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:40.183851 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:40.184520 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:40.184982 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:40.185611 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:40.186919 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:40.188385 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:40.189549 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:40.189749 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:40.191043 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:40.191050 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:40.191063 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:40.191606 (+ 543us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:40.191685 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:40.191706 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:40.191805 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:40.192046 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 01:21:40.192398 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":347,"threads_started":1}
I20260925 01:21:40.199685 7230 raft_consensus.cc:348] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51d17a9f34e44f38d65cad61e17782c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44783 } }
I20260925 01:21:40.200081 7230 raft_consensus.cc:374] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:40.200213 7230 raft_consensus.cc:729] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b51d17a9f34e44f38d65cad61e17782c, State: Initialized, Role: FOLLOWER
I20260925 01:21:40.200922 7230 consensus_queue.cc:260] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51d17a9f34e44f38d65cad61e17782c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44783 } }
I20260925 01:21:40.201181 7230 raft_consensus.cc:388] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:21:40.201313 7230 raft_consensus.cc:482] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:21:40.201498 7230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:40.203873 7230 raft_consensus.cc:504] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51d17a9f34e44f38d65cad61e17782c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44783 } }
I20260925 01:21:40.204422 7230 leader_election.cc:302] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b51d17a9f34e44f38d65cad61e17782c; no voters:
I20260925 01:21:40.205076 7230 leader_election.cc:288] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:21:40.205259 7235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:40.206008 7235 raft_consensus.cc:686] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [term 1 LEADER]: Becoming Leader. State: Replica: b51d17a9f34e44f38d65cad61e17782c, State: Running, Role: LEADER
I20260925 01:21:40.206538 7235 consensus_queue.cc:237] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51d17a9f34e44f38d65cad61e17782c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44783 } }
I20260925 01:21:40.207266 7230 sys_catalog.cc:564] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:21:40.209466 7237 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b51d17a9f34e44f38d65cad61e17782c. Latest consensus state: current_term: 1 leader_uuid: "b51d17a9f34e44f38d65cad61e17782c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51d17a9f34e44f38d65cad61e17782c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44783 } } }
I20260925 01:21:40.209862 7237 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:40.213299 7236 sys_catalog.cc:455] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b51d17a9f34e44f38d65cad61e17782c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b51d17a9f34e44f38d65cad61e17782c" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44783 } } }
I20260925 01:21:40.213749 7236 sys_catalog.cc:458] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:40.214816 7241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:21:40.220235 7241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:21:40.230168 7241 catalog_manager.cc:1347] Generated new cluster ID: 7ca25739b09c45ccb0ddd9e709876a40
I20260925 01:21:40.230296 7241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:21:40.253880 7241 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:21:40.255822 7241 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:21:40.280184 7241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c: Generated new TSK 0
I20260925 01:21:40.281432 7241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:21:40.291429 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:44783
--builtin_ntp_servers=127.20.102.148:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:21:40.630191 7254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:40.630466 7254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:40.630609 7254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:40.638103 7254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:40.638208 7254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:40.638405 7254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:21:40.647608 7254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:44783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:40.648918 7254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:40.650720 7254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:40.663924 7259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:40.663924 7262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:40.663965 7260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:40.665235 7254 server_base.cc:1034] running on GCE node
I20260925 01:21:40.665850 7254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:40.666803 7254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:40.668041 7254 hybrid_clock.cc:648] HybridClock initialized: now 1790299300667996 us; error 43 us; skew 500 ppm
I20260925 01:21:40.668370 7254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:21:40.670746 7254 webserver.cc:460] Webserver started at http://127.20.102.129:46827/ using document root <none> and password file <none>
I20260925 01:21:40.671458 7254 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:40.671597 7254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:40.671931 7254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:40.674932 7254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "65ecc8f8d5d84bc087025aa3ab9b4646"
format_stamp: "Formatted at 2026-09-25 01:21:40 on dist-test-slave-s12s"
server_key: "67b65666c2eba09b3ca0af2b58e6db34"
server_key_iv: "21c7de625a3fe2520ce9dfef6dd0707a"
server_key_version: "encryptionkey@0"
I20260925 01:21:40.675750 7254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "65ecc8f8d5d84bc087025aa3ab9b4646"
format_stamp: "Formatted at 2026-09-25 01:21:40 on dist-test-slave-s12s"
server_key: "67b65666c2eba09b3ca0af2b58e6db34"
server_key_iv: "21c7de625a3fe2520ce9dfef6dd0707a"
server_key_version: "encryptionkey@0"
I20260925 01:21:40.680920 7254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:21:40.684373 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:40.685829 7254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:21:40.686005 7254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "65ecc8f8d5d84bc087025aa3ab9b4646"
format_stamp: "Formatted at 2026-09-25 01:21:40 on dist-test-slave-s12s"
server_key: "67b65666c2eba09b3ca0af2b58e6db34"
server_key_iv: "21c7de625a3fe2520ce9dfef6dd0707a"
server_key_version: "encryptionkey@0"
I20260925 01:21:40.686244 7254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:40.708118 7254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:40.709093 7254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:40.709410 7254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:40.710397 7254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:40.711866 7254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:40.711989 7254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:40.712126 7254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:40.712222 7254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:40.746598 7254 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:41157
I20260925 01:21:40.746704 7380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:41157 every 8 connection(s)
I20260925 01:21:40.747980 7254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:21:40.757401 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 7254
I20260925 01:21:40.757599 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:21:40.758067 20890 external_mini_cluster.cc:1428] Setting key 4d9c7c4ce8c18ab1168a850172ccf11e
I20260925 01:21:40.761001 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:44783
--builtin_ntp_servers=127.20.102.148:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:21:40.762203 7383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:40.751213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:37687 (local address 127.20.102.190:44783)
0925 01:21:40.751746 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:40.751753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:40.753183 (+ 1430us) server_negotiation.cc:407] Connection header received
0925 01:21:40.755052 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:40.755058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:40.755159 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:40.755311 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:40.756407 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:40.757394 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:40.759182 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:40.759679 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:40.759797 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:40.761327 (+ 1530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:40.761332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:40.761336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:40.761354 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:40.761397 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:40.761409 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:40.761523 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:40.761757 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 01:21:40.761859 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":247,"threads_started":1}
I20260925 01:21:40.763266 7384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:40.751837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:44783 (local address 127.20.102.129:37687)
0925 01:21:40.752962 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:40.753053 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:40.754774 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:40.755548 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:40.755556 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:40.755645 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:40.756234 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:40.756249 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:40.757586 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:40.757592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:40.759021 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:40.759037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:40.759914 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:40.759924 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:40.760196 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:40.760962 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:21:40.760965 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:40.761005 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:21:40.761119 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:40.761522 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:40.761534 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:40.761546 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 01:21:40.761744 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 01:21:40.762152 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":321,"threads_started":1}
I20260925 01:21:40.765066 7381 heartbeater.cc:347] Connected to a master server at 127.20.102.190:44783
I20260925 01:21:40.765416 7381 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:40.766388 7381 heartbeater.cc:511] Master 127.20.102.190:44783 requested a full tablet report, sending...
I20260925 01:21:40.768579 7195 ts_manager.cc:194] Registered new tserver with Master: 65ecc8f8d5d84bc087025aa3ab9b4646 (127.20.102.129:41157)
I20260925 01:21:40.770148 7195 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.102.129:37687
W20260925 01:21:41.099619 7385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:41.099897 7385 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:41.100054 7385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:41.107381 7385 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:41.107475 7385 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:41.107679 7385 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:21:41.117002 7385 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:44783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:41.118342 7385 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:41.120126 7385 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:41.133245 7391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:41.133296 7390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:41.133194 7393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:41.142606 7385 server_base.cc:1034] running on GCE node
I20260925 01:21:41.143347 7385 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:41.144388 7385 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:41.145618 7385 hybrid_clock.cc:648] HybridClock initialized: now 1790299301145529 us; error 85 us; skew 500 ppm
I20260925 01:21:41.145925 7385 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:21:41.148176 7385 webserver.cc:460] Webserver started at http://127.20.102.130:36155/ using document root <none> and password file <none>
I20260925 01:21:41.148989 7385 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:41.149092 7385 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:41.149355 7385 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:41.152220 7385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "dde5ae4b2a3a47ba8a09410eded5a830"
format_stamp: "Formatted at 2026-09-25 01:21:41 on dist-test-slave-s12s"
server_key: "2ed997ab24049aa6784cd3c0335146b4"
server_key_iv: "243fc9eb4856b612f572cc20e6c0e04a"
server_key_version: "encryptionkey@0"
I20260925 01:21:41.153021 7385 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "dde5ae4b2a3a47ba8a09410eded5a830"
format_stamp: "Formatted at 2026-09-25 01:21:41 on dist-test-slave-s12s"
server_key: "2ed997ab24049aa6784cd3c0335146b4"
server_key_iv: "243fc9eb4856b612f572cc20e6c0e04a"
server_key_version: "encryptionkey@0"
I20260925 01:21:41.158056 7385 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260925 01:21:41.161557 7399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:41.162904 7385 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:21:41.163071 7385 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "dde5ae4b2a3a47ba8a09410eded5a830"
format_stamp: "Formatted at 2026-09-25 01:21:41 on dist-test-slave-s12s"
server_key: "2ed997ab24049aa6784cd3c0335146b4"
server_key_iv: "243fc9eb4856b612f572cc20e6c0e04a"
server_key_version: "encryptionkey@0"
I20260925 01:21:41.163278 7385 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:41.192325 7385 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:41.193281 7385 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:41.193601 7385 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:41.194664 7385 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:41.196199 7385 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:41.196323 7385 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:41.196477 7385 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:41.196570 7385 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:41.231297 7385 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:42945
I20260925 01:21:41.231392 7511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:42945 every 8 connection(s)
I20260925 01:21:41.232692 7385 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:21:41.237537 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 7385
I20260925 01:21:41.237740 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:21:41.238348 20890 external_mini_cluster.cc:1428] Setting key 04f3bd810e2eb08c5266f9ea197b6c9e
I20260925 01:21:41.241617 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:44783
--builtin_ntp_servers=127.20.102.148:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:21:41.247977 7383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.235410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:55235 (local address 127.20.102.190:44783)
0925 01:21:41.235629 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:41.235634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:41.237242 (+ 1608us) server_negotiation.cc:407] Connection header received
0925 01:21:41.239719 (+ 2477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:41.239725 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:41.239818 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:41.239937 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:41.241436 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.242305 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.244929 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.245347 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.245452 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.247171 (+ 1719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:41.247176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:41.247179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:41.247200 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:41.247232 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:41.247244 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:41.247309 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:41.247658 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 01:21:41.247746 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 01:21:41.249526 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.236009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:44783 (local address 127.20.102.130:55235)
0925 01:21:41.237011 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:41.237060 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:41.239427 (+ 2367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:41.240221 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:41.240235 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:41.240358 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:41.241232 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:41.241262 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.242477 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.242486 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:41.244761 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:41.244780 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.245585 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.245594 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:41.245929 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.246782 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:21:41.246788 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:41.246846 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:21:41.246953 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:41.247344 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:41.247357 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:41.247369 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 01:21:41.247561 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 01:21:41.248017 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":284,"threads_started":1}
I20260925 01:21:41.251210 7512 heartbeater.cc:347] Connected to a master server at 127.20.102.190:44783
I20260925 01:21:41.251585 7512 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:41.252378 7512 heartbeater.cc:511] Master 127.20.102.190:44783 requested a full tablet report, sending...
I20260925 01:21:41.254076 7195 ts_manager.cc:194] Registered new tserver with Master: dde5ae4b2a3a47ba8a09410eded5a830 (127.20.102.130:42945)
I20260925 01:21:41.255332 7195 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.102.130:55235
W20260925 01:21:41.580060 7515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:41.580349 7515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:41.580437 7515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:41.587689 7515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:41.587797 7515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:41.587906 7515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:21:41.599973 7515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:44783
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:41.601775 7515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:41.603941 7515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:41.617789 7520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:41.617795 7521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:41.617861 7523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:41.618422 7515 server_base.cc:1034] running on GCE node
I20260925 01:21:41.619097 7515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:41.620028 7515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:41.621271 7515 hybrid_clock.cc:648] HybridClock initialized: now 1790299301621193 us; error 56 us; skew 500 ppm
I20260925 01:21:41.621706 7515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:21:41.629490 7515 webserver.cc:460] Webserver started at http://127.20.102.131:35517/ using document root <none> and password file <none>
I20260925 01:21:41.630272 7515 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:41.630391 7515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:41.630736 7515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:41.634171 7515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "bf99ad7a44074215adbfe515e00f6066"
format_stamp: "Formatted at 2026-09-25 01:21:41 on dist-test-slave-s12s"
server_key: "c1ebc640263e5a4b591b08b64f50aa85"
server_key_iv: "6590bfa435b33ebf96e8b27deced4360"
server_key_version: "encryptionkey@0"
I20260925 01:21:41.635053 7515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "bf99ad7a44074215adbfe515e00f6066"
format_stamp: "Formatted at 2026-09-25 01:21:41 on dist-test-slave-s12s"
server_key: "c1ebc640263e5a4b591b08b64f50aa85"
server_key_iv: "6590bfa435b33ebf96e8b27deced4360"
server_key_version: "encryptionkey@0"
I20260925 01:21:41.640754 7515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:21:41.644469 7529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:41.645922 7515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 01:21:41.646119 7515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "bf99ad7a44074215adbfe515e00f6066"
format_stamp: "Formatted at 2026-09-25 01:21:41 on dist-test-slave-s12s"
server_key: "c1ebc640263e5a4b591b08b64f50aa85"
server_key_iv: "6590bfa435b33ebf96e8b27deced4360"
server_key_version: "encryptionkey@0"
I20260925 01:21:41.646361 7515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:41.662405 7515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:41.663211 7515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:41.663538 7515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:41.664485 7515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:41.666076 7515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:41.666198 7515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:41.666324 7515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:41.666424 7515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:41.699754 7515 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:33057
I20260925 01:21:41.699801 7641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:33057 every 8 connection(s)
I20260925 01:21:41.701354 7515 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:21:41.710479 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 7515
I20260925 01:21:41.710667 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:21:41.711174 20890 external_mini_cluster.cc:1428] Setting key ebc1ec6a0c1470617331229c657a80af
I20260925 01:21:41.714318 7383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.704027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:58229 (local address 127.20.102.190:44783)
0925 01:21:41.704260 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:41.704266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:41.705743 (+ 1477us) server_negotiation.cc:407] Connection header received
0925 01:21:41.707461 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:41.707469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:41.707552 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:41.707642 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:41.708837 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.709573 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.711171 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.711768 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.711914 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.713374 (+ 1460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:41.713380 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:41.713385 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:41.713411 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:41.713468 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:41.713484 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:41.713602 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:41.713838 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 01:21:41.714004 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 01:21:41.715332 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.704597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:44783 (local address 127.20.102.131:58229)
0925 01:21:41.705554 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:41.705596 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:41.707176 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:41.707932 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:41.707941 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:41.708029 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:41.708628 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:41.708644 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.709698 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.709702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:41.711023 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:41.711035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.712005 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.712014 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:41.712260 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.713049 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:21:41.713051 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:41.713090 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:21:41.713164 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:41.713600 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:41.713612 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:41.713621 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 01:21:41.713801 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 01:21:41.714198 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":293,"threads_started":1}
I20260925 01:21:41.716602 7642 heartbeater.cc:347] Connected to a master server at 127.20.102.190:44783
I20260925 01:21:41.716955 7642 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:41.717681 7642 heartbeater.cc:511] Master 127.20.102.190:44783 requested a full tablet report, sending...
I20260925 01:21:41.719002 7195 ts_manager.cc:194] Registered new tserver with Master: bf99ad7a44074215adbfe515e00f6066 (127.20.102.131:33057)
I20260925 01:21:41.719791 7195 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.102.131:58229
I20260925 01:21:41.726348 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:21:41.736241 7383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.730572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33014 (local address 127.20.102.190:44783)
0925 01:21:41.730733 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:41.730737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:41.731189 (+ 452us) server_negotiation.cc:407] Connection header received
0925 01:21:41.731319 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:41.731322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:41.731383 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:41.731458 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:41.731884 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.732592 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.733972 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.734404 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.734536 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.735491 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:41.735496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:41.735499 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:41.735515 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:41.735544 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:41.735583 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:41.735701 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:41.735858 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 01:21:41.735932 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:21:41.740010 7195 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:21:41.747144 7195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:21:41.773219 7381 heartbeater.cc:503] Master 127.20.102.190:44783 was elected leader, sending a full tablet report...
I20260925 01:21:41.786026 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.760573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58330 (local address 127.20.102.130:42945)
0925 01:21:41.779109 (+ 18536us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:41.779119 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:41.779151 (+ 32us) server_negotiation.cc:407] Connection header received
0925 01:21:41.779259 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:41.779267 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:41.779488 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:41.779713 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:41.780438 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.781471 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.783784 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.784829 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.785013 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.785280 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:41.785489 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 01:21:41.785664 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18379,"thread_start_us":18270,"threads_started":1}
I20260925 01:21:41.786111 7655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.760457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:42945 (local address 127.0.0.1:58330)
0925 01:21:41.775069 (+ 14612us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:41.775095 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:41.775269 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:41.779722 (+ 4453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:41.779726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:41.779752 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:41.780223 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:41.780235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.781630 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.781638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:41.783584 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:41.783602 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.785015 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.785021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:41.785219 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.785270 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 01:21:41.785435 (+ 165us) client_negotiation.cc:240] Negotiation successful
0925 01:21:41.785653 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14433,"mutex_wait_us":26,"thread_start_us":15378,"threads_started":1}
I20260925 01:21:41.786944 7657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.760441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:33057 (local address 127.0.0.1:45080)
0925 01:21:41.772117 (+ 11676us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:41.772183 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:41.772416 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:41.776370 (+ 3954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:41.776383 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:41.776434 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:41.777003 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:41.777026 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.779896 (+ 2870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.779902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:41.783262 (+ 3360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:41.783280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.785987 (+ 2707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.785993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:41.786233 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.786287 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 01:21:41.786373 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 01:21:41.786510 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11317,"mutex_wait_us":149,"thread_start_us":14221,"threads_started":1}
I20260925 01:21:41.786984 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.761805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45080 (local address 127.20.102.131:33057)
0925 01:21:41.774698 (+ 12893us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:41.774712 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:41.774757 (+ 45us) server_negotiation.cc:407] Connection header received
0925 01:21:41.774873 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:41.774882 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:41.775276 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:41.775538 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:41.777898 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.779707 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.783627 (+ 3920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.785331 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.785554 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.785787 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:41.786452 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 01:21:41.786632 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12655,"thread_start_us":12464,"threads_started":1}
I20260925 01:21:41.787912 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.760989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:41157 (local address 127.0.0.1:38578)
0925 01:21:41.775795 (+ 14806us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:41.775818 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:41.775939 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:41.778208 (+ 2269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:41.778213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:41.778237 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:41.778709 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:41.778726 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.780360 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.780367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:41.783262 (+ 2895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:41.783280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.785970 (+ 2690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.785979 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:41.786742 (+ 763us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.786779 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 01:21:41.786889 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 01:21:41.787577 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14659,"mutex_wait_us":131,"thread_start_us":10613,"threads_started":1}
I20260925 01:21:41.788766 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.761310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38578 (local address 127.20.102.129:41157)
0925 01:21:41.777301 (+ 15991us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:41.777314 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:41.777352 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:21:41.777465 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:41.777476 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:41.777760 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:41.777957 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:41.778934 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.780211 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.783625 (+ 3414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.785331 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.785566 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.785787 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:41.788166 (+ 2379us) server_negotiation.cc:299] Negotiation successful
0925 01:21:41.788363 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15748,"thread_start_us":15582,"threads_started":1}
I20260925 01:21:41.790576 7447 tablet_service.cc:1467] Processing CreateTablet for tablet 5abcf11b2976491999457b5e239015c1 (DEFAULT_TABLE table=test-table [id=93168d2e32b14fa18281a5db172db1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:41.790987 7577 tablet_service.cc:1467] Processing CreateTablet for tablet 5abcf11b2976491999457b5e239015c1 (DEFAULT_TABLE table=test-table [id=93168d2e32b14fa18281a5db172db1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:41.792160 7447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5abcf11b2976491999457b5e239015c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:41.792387 7577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5abcf11b2976491999457b5e239015c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:41.792348 7316 tablet_service.cc:1467] Processing CreateTablet for tablet 5abcf11b2976491999457b5e239015c1 (DEFAULT_TABLE table=test-table [id=93168d2e32b14fa18281a5db172db1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:41.793740 7316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5abcf11b2976491999457b5e239015c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:41.806762 7660 tablet_bootstrap.cc:492] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066: Bootstrap starting.
I20260925 01:21:41.809170 7661 tablet_bootstrap.cc:492] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830: Bootstrap starting.
I20260925 01:21:41.811247 7660 tablet_bootstrap.cc:654] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:41.812920 7660 log.cc:824] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:41.813231 7661 tablet_bootstrap.cc:654] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:41.814525 7662 tablet_bootstrap.cc:492] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646: Bootstrap starting.
I20260925 01:21:41.817498 7661 log.cc:824] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:41.819090 7662 tablet_bootstrap.cc:654] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:41.820431 7662 log.cc:824] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:41.820887 7661 tablet_bootstrap.cc:492] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830: No bootstrap required, opened a new log
I20260925 01:21:41.821002 7660 tablet_bootstrap.cc:492] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066: No bootstrap required, opened a new log
I20260925 01:21:41.821223 7661 ts_tablet_manager.cc:1397] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260925 01:21:41.821336 7660 ts_tablet_manager.cc:1397] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.007s
I20260925 01:21:41.823815 7662 tablet_bootstrap.cc:492] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646: No bootstrap required, opened a new log
I20260925 01:21:41.824108 7662 ts_tablet_manager.cc:1397] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 01:21:41.829581 7661 raft_consensus.cc:348] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde5ae4b2a3a47ba8a09410eded5a830" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42945 } } peers { permanent_uuid: "bf99ad7a44074215adbfe515e00f6066" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33057 } } peers { permanent_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41157 } }
I20260925 01:21:41.829996 7661 raft_consensus.cc:374] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:41.830111 7661 raft_consensus.cc:729] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dde5ae4b2a3a47ba8a09410eded5a830, State: Initialized, Role: FOLLOWER
I20260925 01:21:41.830751 7661 consensus_queue.cc:260] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde5ae4b2a3a47ba8a09410eded5a830" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42945 } } peers { permanent_uuid: "bf99ad7a44074215adbfe515e00f6066" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33057 } } peers { permanent_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41157 } }
I20260925 01:21:41.831894 7512 heartbeater.cc:503] Master 127.20.102.190:44783 was elected leader, sending a full tablet report...
I20260925 01:21:41.832428 7661 ts_tablet_manager.cc:1428] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260925 01:21:41.834450 7660 raft_consensus.cc:348] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde5ae4b2a3a47ba8a09410eded5a830" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42945 } } peers { permanent_uuid: "bf99ad7a44074215adbfe515e00f6066" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33057 } } peers { permanent_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41157 } }
I20260925 01:21:41.834909 7660 raft_consensus.cc:374] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:41.835011 7660 raft_consensus.cc:729] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf99ad7a44074215adbfe515e00f6066, State: Initialized, Role: FOLLOWER
I20260925 01:21:41.835112 7662 raft_consensus.cc:348] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde5ae4b2a3a47ba8a09410eded5a830" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42945 } } peers { permanent_uuid: "bf99ad7a44074215adbfe515e00f6066" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33057 } } peers { permanent_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41157 } }
I20260925 01:21:41.835522 7662 raft_consensus.cc:374] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:41.835649 7662 raft_consensus.cc:729] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ecc8f8d5d84bc087025aa3ab9b4646, State: Initialized, Role: FOLLOWER
I20260925 01:21:41.835764 7660 consensus_queue.cc:260] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde5ae4b2a3a47ba8a09410eded5a830" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42945 } } peers { permanent_uuid: "bf99ad7a44074215adbfe515e00f6066" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33057 } } peers { permanent_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41157 } }
I20260925 01:21:41.836351 7662 consensus_queue.cc:260] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde5ae4b2a3a47ba8a09410eded5a830" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42945 } } peers { permanent_uuid: "bf99ad7a44074215adbfe515e00f6066" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33057 } } peers { permanent_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41157 } }
I20260925 01:21:41.838558 7662 ts_tablet_manager.cc:1428] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646: Time spent starting tablet: real 0.014s user 0.008s sys 0.008s
I20260925 01:21:41.839757 7642 heartbeater.cc:503] Master 127.20.102.190:44783 was elected leader, sending a full tablet report...
I20260925 01:21:41.840385 7660 ts_tablet_manager.cc:1428] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066: Time spent starting tablet: real 0.019s user 0.011s sys 0.005s
I20260925 01:21:41.885721 7668 raft_consensus.cc:482] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:21:41.886096 7668 raft_consensus.cc:504] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde5ae4b2a3a47ba8a09410eded5a830" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42945 } } peers { permanent_uuid: "bf99ad7a44074215adbfe515e00f6066" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33057 } } peers { permanent_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41157 } }
I20260925 01:21:41.887928 7668 leader_election.cc:288] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dde5ae4b2a3a47ba8a09410eded5a830 (127.20.102.130:42945), bf99ad7a44074215adbfe515e00f6066 (127.20.102.131:33057)
I20260925 01:21:41.895402 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.888511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:49739 (local address 127.20.102.130:42945)
0925 01:21:41.888777 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:41.888785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:41.889594 (+ 809us) server_negotiation.cc:407] Connection header received
0925 01:21:41.889851 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:41.889858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:41.889942 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:41.890042 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:41.890760 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.891560 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.893626 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.894378 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.894553 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.894626 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:41.894973 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 01:21:41.895122 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:21:41.895403 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.888526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:36109 (local address 127.20.102.131:33057)
0925 01:21:41.888778 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:41.888784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:41.889730 (+ 946us) server_negotiation.cc:407] Connection header received
0925 01:21:41.889954 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:41.889966 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:41.890068 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:41.890176 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:41.890885 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.891591 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.893633 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.894421 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.894596 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.894666 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:41.894995 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 01:21:41.895139 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:21:41.895514 7670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.888537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:33057 (local address 127.20.102.129:36109)
0925 01:21:41.889603 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:41.889623 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:41.889796 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:41.890201 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:41.890205 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:41.890244 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:41.890711 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:41.890721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.891728 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.891735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:41.893419 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:41.893431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.894622 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.894628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:41.894839 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.894863 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:21:41.894967 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 01:21:41.895141 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"mutex_wait_us":214,"thread_start_us":475,"threads_started":1}
I20260925 01:21:41.895562 7669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.888518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:42945 (local address 127.20.102.129:49739)
0925 01:21:41.889476 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:41.889492 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:41.889649 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:41.890056 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:41.890061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:41.890085 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:41.890597 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:41.890618 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.891698 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.891705 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:41.893411 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:41.893423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.894580 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.894586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:41.894795 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.894836 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 01:21:41.894944 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 01:21:41.895127 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":110,"thread_start_us":564,"threads_started":1}
I20260925 01:21:41.896821 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5abcf11b2976491999457b5e239015c1" candidate_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dde5ae4b2a3a47ba8a09410eded5a830" is_pre_election: true
I20260925 01:21:41.896824 7597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5abcf11b2976491999457b5e239015c1" candidate_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf99ad7a44074215adbfe515e00f6066" is_pre_election: true
I20260925 01:21:41.897341 7467 raft_consensus.cc:2454] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65ecc8f8d5d84bc087025aa3ab9b4646 in term 0.
I20260925 01:21:41.897347 7597 raft_consensus.cc:2454] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65ecc8f8d5d84bc087025aa3ab9b4646 in term 0.
I20260925 01:21:41.898030 7272 leader_election.cc:302] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65ecc8f8d5d84bc087025aa3ab9b4646, bf99ad7a44074215adbfe515e00f6066; no voters:
I20260925 01:21:41.898561 7668 raft_consensus.cc:2781] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:21:41.898742 7668 raft_consensus.cc:482] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:21:41.898871 7668 raft_consensus.cc:3037] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:41.901254 7668 raft_consensus.cc:504] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde5ae4b2a3a47ba8a09410eded5a830" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42945 } } peers { permanent_uuid: "bf99ad7a44074215adbfe515e00f6066" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33057 } } peers { permanent_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41157 } }
I20260925 01:21:41.901865 7668 leader_election.cc:288] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [CANDIDATE]: Term 1 election: Requested vote from peers dde5ae4b2a3a47ba8a09410eded5a830 (127.20.102.130:42945), bf99ad7a44074215adbfe515e00f6066 (127.20.102.131:33057)
I20260925 01:21:41.902403 7597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5abcf11b2976491999457b5e239015c1" candidate_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf99ad7a44074215adbfe515e00f6066"
I20260925 01:21:41.902403 7467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5abcf11b2976491999457b5e239015c1" candidate_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dde5ae4b2a3a47ba8a09410eded5a830"
I20260925 01:21:41.902622 7467 raft_consensus.cc:3037] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:41.902622 7597 raft_consensus.cc:3037] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:41.905937 7467 raft_consensus.cc:2454] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65ecc8f8d5d84bc087025aa3ab9b4646 in term 1.
I20260925 01:21:41.905937 7597 raft_consensus.cc:2454] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65ecc8f8d5d84bc087025aa3ab9b4646 in term 1.
I20260925 01:21:41.906387 7271 leader_election.cc:302] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65ecc8f8d5d84bc087025aa3ab9b4646, dde5ae4b2a3a47ba8a09410eded5a830; no voters:
I20260925 01:21:41.906718 7668 raft_consensus.cc:2781] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:41.907145 7668 raft_consensus.cc:686] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [term 1 LEADER]: Becoming Leader. State: Replica: 65ecc8f8d5d84bc087025aa3ab9b4646, State: Running, Role: LEADER
I20260925 01:21:41.907672 7668 consensus_queue.cc:237] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde5ae4b2a3a47ba8a09410eded5a830" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42945 } } peers { permanent_uuid: "bf99ad7a44074215adbfe515e00f6066" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33057 } } peers { permanent_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41157 } }
I20260925 01:21:41.912643 7195 catalog_manager.cc:5495] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65ecc8f8d5d84bc087025aa3ab9b4646 (127.20.102.129). New cstate: current_term: 1 leader_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dde5ae4b2a3a47ba8a09410eded5a830" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf99ad7a44074215adbfe515e00f6066" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65ecc8f8d5d84bc087025aa3ab9b4646" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 41157 } health_report { overall_health: HEALTHY } } }
I20260925 01:21:41.942591 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.936999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.20.102.129:41157)
0925 01:21:41.937220 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:41.937226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:41.937246 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:21:41.937472 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:41.937477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:41.937613 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:41.937774 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:21:41.938166 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.938860 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.940226 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:41.940655 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:41.940848 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:41.941480 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:21:41.941987 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:21:41.942178 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:41.942306 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 01:21:41.942351 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:21:41.951344 7467 raft_consensus.cc:1261] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830 [term 1 FOLLOWER]: Refusing update from remote peer 65ecc8f8d5d84bc087025aa3ab9b4646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:21:41.951371 7597 raft_consensus.cc:1261] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066 [term 1 FOLLOWER]: Refusing update from remote peer 65ecc8f8d5d84bc087025aa3ab9b4646: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:21:41.952374 7671 consensus_queue.cc:1035] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dde5ae4b2a3a47ba8a09410eded5a830" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 42945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:21:41.953013 7668 consensus_queue.cc:1035] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf99ad7a44074215adbfe515e00f6066" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 33057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 01:21:41.953111 7643 tablet.cc:2369] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:21:41.981375 7193 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:21:41.981693 7193 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:21:41.986052 7193 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b51d17a9f34e44f38d65cad61e17782c: Sending DeleteTablet for 3 replicas of tablet 5abcf11b2976491999457b5e239015c1
I20260925 01:21:41.987332 7447 tablet_service.cc:1514] Processing DeleteTablet for tablet 5abcf11b2976491999457b5e239015c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:41 UTC) from {username='slave'} at 127.0.0.1:58330
I20260925 01:21:41.987417 7577 tablet_service.cc:1514] Processing DeleteTablet for tablet 5abcf11b2976491999457b5e239015c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:41 UTC) from {username='slave'} at 127.0.0.1:45080
I20260925 01:21:41.987429 7316 tablet_service.cc:1514] Processing DeleteTablet for tablet 5abcf11b2976491999457b5e239015c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:41 UTC) from {username='slave'} at 127.0.0.1:38578
I20260925 01:21:41.991187 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 7254
I20260925 01:21:42.000562 7685 tablet_replica.cc:330] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066: stopping tablet replica
I20260925 01:21:42.001325 7685 raft_consensus.cc:2229] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:21:42.002017 7685 raft_consensus.cc:2258] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:42.003021 7686 tablet_replica.cc:330] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646: stopping tablet replica
I20260925 01:21:42.003620 7686 raft_consensus.cc:2229] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:21:42.003966 7684 tablet_replica.cc:330] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830: stopping tablet replica
I20260925 01:21:42.004432 7684 raft_consensus.cc:2229] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:21:42.004626 7686 raft_consensus.cc:2258] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:42.005363 7684 raft_consensus.cc:2258] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:42.007833 7686 ts_tablet_manager.cc:1905] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:42.008049 7684 ts_tablet_manager.cc:1905] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:42.010057 7685 ts_tablet_manager.cc:1905] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:42.017323 7686 ts_tablet_manager.cc:1918] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:42.017606 7686 log.cc:1196] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/5abcf11b2976491999457b5e239015c1
I20260925 01:21:42.017850 7685 ts_tablet_manager.cc:1918] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:42.018052 7685 log.cc:1196] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/5abcf11b2976491999457b5e239015c1
I20260925 01:21:42.018462 7686 ts_tablet_manager.cc:1939] T 5abcf11b2976491999457b5e239015c1 P 65ecc8f8d5d84bc087025aa3ab9b4646: Deleting consensus metadata
I20260925 01:21:42.018680 7685 ts_tablet_manager.cc:1939] T 5abcf11b2976491999457b5e239015c1 P bf99ad7a44074215adbfe515e00f6066: Deleting consensus metadata
I20260925 01:21:42.020597 7180 catalog_manager.cc:4841] TS 65ecc8f8d5d84bc087025aa3ab9b4646 (127.20.102.129:41157): tablet 5abcf11b2976491999457b5e239015c1 (table test-table [id=93168d2e32b14fa18281a5db172db1e2]) successfully deleted
I20260925 01:21:42.020620 7182 catalog_manager.cc:4841] TS bf99ad7a44074215adbfe515e00f6066 (127.20.102.131:33057): tablet 5abcf11b2976491999457b5e239015c1 (table test-table [id=93168d2e32b14fa18281a5db172db1e2]) successfully deleted
I20260925 01:21:42.022007 7684 ts_tablet_manager.cc:1918] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:42.022214 7684 log.cc:1196] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/5abcf11b2976491999457b5e239015c1
I20260925 01:21:42.022805 7684 ts_tablet_manager.cc:1939] T 5abcf11b2976491999457b5e239015c1 P dde5ae4b2a3a47ba8a09410eded5a830: Deleting consensus metadata
I20260925 01:21:42.023437 7658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:41.992841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54528 (local address 127.20.102.129:41157)
0925 01:21:41.993025 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:41.993031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:42.008081 (+ 15050us) server_negotiation.cc:407] Connection header received
0925 01:21:42.008342 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:42.008349 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:42.008447 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:42.008593 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:42.009197 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:42.010544 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:42.012664 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:42.013428 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:42.013553 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:42.021577 (+ 8024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:42.021590 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:42.021608 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:42.022493 (+ 885us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:42.022566 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:42.022593 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:42.022725 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:42.022950 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 01:21:42.023071 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:21:42.024328 7376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:42.024901 7181 catalog_manager.cc:4841] TS dde5ae4b2a3a47ba8a09410eded5a830 (127.20.102.130:42945): tablet 5abcf11b2976491999457b5e239015c1 (table test-table [id=93168d2e32b14fa18281a5db172db1e2]) successfully deleted
I20260925 01:21:42.123840 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 7254
I20260925 01:21:42.143016 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 7385
I20260925 01:21:42.150516 7656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:42.143928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49866 (local address 127.20.102.130:42945)
0925 01:21:42.144113 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:42.144117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:42.144135 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:21:42.144215 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:42.144219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:42.144317 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:42.144401 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:42.145114 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:42.145912 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:42.147363 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:42.147783 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:42.147876 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:42.149025 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:42.149033 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:42.149048 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:42.149677 (+ 629us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:42.149724 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:42.149750 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:42.149910 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:42.150031 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 01:21:42.150169 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:21:42.151333 7507 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:42.240139 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 7385
I20260925 01:21:42.258224 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 7515
I20260925 01:21:42.265230 7659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:42.259039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60788 (local address 127.20.102.131:33057)
0925 01:21:42.259240 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:42.259246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:42.259266 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:21:42.259440 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:42.259446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:42.259541 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:42.259652 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:42.260101 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:42.260906 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:42.262179 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:42.262579 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:42.262710 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:42.263813 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:42.263822 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:42.263838 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:42.264412 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:42.264459 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:42.264477 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:42.264598 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:42.264819 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 01:21:42.264923 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:21:42.265869 7637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:42.357383 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 7515
I20260925 01:21:42.374782 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 7164
I20260925 01:21:42.375952 7225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:42.485423 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 7164
2026-09-25T01:21:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2828 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:21:42 dist-test-slave-s12s krb5kdc[7694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:21:42 dist-test-slave-s12s krb5kdc[7694](info): set up 2 sockets
Sep 25 01:21:42 dist-test-slave-s12s krb5kdc[7694](info): commencing operation
krb5kdc: starting...
W20260925 01:21:43.605615 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:21:43 dist-test-slave-s12s krb5kdc[7694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:21:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:21:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:21:43.863837 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:35351
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:35351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:44.213867 7707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:44.214210 7707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:44.214382 7707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:44.221678 7707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:21:44.221830 7707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:44.221943 7707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:44.222054 7707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:21:44.222146 7707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:21:44.232009 7707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:35351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:35351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:44.233377 7707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:44.234727 7707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:44.246740 7712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:44.246793 7713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:44.247052 7715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:44.247915 7707 server_base.cc:1034] running on GCE node
I20260925 01:21:44.248798 7707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:44.250270 7707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:44.251542 7707 hybrid_clock.cc:648] HybridClock initialized: now 1790299304251447 us; error 38 us; skew 500 ppm
I20260925 01:21:44.251935 7707 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:44 dist-test-slave-s12s krb5kdc[7694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299304, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:44.255209 7707 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:21:44.256877 7707 webserver.cc:460] Webserver started at http://127.20.102.190:45799/ using document root <none> and password file <none>
I20260925 01:21:44.257586 7707 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:44.257723 7707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:44.258046 7707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:44.261175 7707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "bcbb2bb6d2b74802833d4162e7d1b40f"
format_stamp: "Formatted at 2026-09-25 01:21:44 on dist-test-slave-s12s"
server_key: "96348220b6b85adc293b93a72c65fd10"
server_key_iv: "fef0c783d16233f4eb9c9c0e9f91d296"
server_key_version: "encryptionkey@0"
I20260925 01:21:44.261970 7707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "bcbb2bb6d2b74802833d4162e7d1b40f"
format_stamp: "Formatted at 2026-09-25 01:21:44 on dist-test-slave-s12s"
server_key: "96348220b6b85adc293b93a72c65fd10"
server_key_iv: "fef0c783d16233f4eb9c9c0e9f91d296"
server_key_version: "encryptionkey@0"
I20260925 01:21:44.267169 7707 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:21:44.270648 7722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:44.271921 7707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:21:44.272107 7707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "bcbb2bb6d2b74802833d4162e7d1b40f"
format_stamp: "Formatted at 2026-09-25 01:21:44 on dist-test-slave-s12s"
server_key: "96348220b6b85adc293b93a72c65fd10"
server_key_iv: "fef0c783d16233f4eb9c9c0e9f91d296"
server_key_version: "encryptionkey@0"
I20260925 01:21:44.272322 7707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:44.292006 7707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:44.296233 7707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:44.296531 7707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:44.315883 7707 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:35351
I20260925 01:21:44.315907 7773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:35351 every 8 connection(s)
I20260925 01:21:44.317458 7707 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:21:44.319517 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 7707
I20260925 01:21:44.319751 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:21:44.320226 20890 external_mini_cluster.cc:1428] Setting key bc1ea80a9c9270f60311b98d064fd73a
I20260925 01:21:44.321841 7774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 01:21:44 dist-test-slave-s12s krb5kdc[7694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:21:44.335772 7774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f: Bootstrap starting.
I20260925 01:21:44.339965 7774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:44.341420 7774 log.cc:824] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:44.344981 7774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f: No bootstrap required, opened a new log
I20260925 01:21:44.345320 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:44.322144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.20.102.190:35351)
0925 01:21:44.322927 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:44.322939 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:44.322979 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:21:44.324594 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:44.324615 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:44.325070 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:44.325395 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:44.327305 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:44.328763 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:44.330536 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:44.331204 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:44.331345 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:44.336299 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:44.336309 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:44.336328 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:44.336371 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:44.340619 (+ 4248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:44.341352 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:44.341362 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:44.341372 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:44.341662 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:44.342155 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:44.342160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:44.342163 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:44.342796 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:44.343115 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:44.343521 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 01:21:44.343970 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":379,"threads_started":1}
I20260925 01:21:44.353129 7774 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbb2bb6d2b74802833d4162e7d1b40f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35351 } }
I20260925 01:21:44.353472 7774 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:44.353546 7774 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcbb2bb6d2b74802833d4162e7d1b40f, State: Initialized, Role: FOLLOWER
I20260925 01:21:44.354197 7774 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbb2bb6d2b74802833d4162e7d1b40f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35351 } }
I20260925 01:21:44.354460 7774 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:21:44.354614 7774 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:21:44.354808 7774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:44.357059 7774 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbb2bb6d2b74802833d4162e7d1b40f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35351 } }
I20260925 01:21:44.357589 7774 leader_election.cc:302] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcbb2bb6d2b74802833d4162e7d1b40f; no voters:
I20260925 01:21:44.358225 7774 leader_election.cc:288] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:21:44.358392 7779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:44.359166 7779 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [term 1 LEADER]: Becoming Leader. State: Replica: bcbb2bb6d2b74802833d4162e7d1b40f, State: Running, Role: LEADER
I20260925 01:21:44.359735 7779 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbb2bb6d2b74802833d4162e7d1b40f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35351 } }
I20260925 01:21:44.360509 7774 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:21:44.362581 7780 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcbb2bb6d2b74802833d4162e7d1b40f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbb2bb6d2b74802833d4162e7d1b40f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35351 } } }
I20260925 01:21:44.363075 7780 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:44.363338 7781 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcbb2bb6d2b74802833d4162e7d1b40f. Latest consensus state: current_term: 1 leader_uuid: "bcbb2bb6d2b74802833d4162e7d1b40f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcbb2bb6d2b74802833d4162e7d1b40f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 35351 } } }
I20260925 01:21:44.364274 7781 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:44.365303 7785 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:21:44.370534 7785 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:21:44.380225 7785 catalog_manager.cc:1347] Generated new cluster ID: 0755cfe710da4337841e3b2b66edd04f
I20260925 01:21:44.380342 7785 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:21:44.410171 7785 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:21:44.412261 7785 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:21:44.420120 7785 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f: Generated new TSK 0
I20260925 01:21:44.421236 7785 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.20.102.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
Entry for principal kudu/127.20.102.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab.
I20260925 01:21:44.484227 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35351
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:44.832329 7800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:44.832675 7800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:44.832922 7800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:44.840565 7800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:44.840773 7800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:44.840966 7800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:21:44.855554 7800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.129.keytab
--principal=kudu/127.20.102.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:44.857049 7800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:44.858556 7800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:44.872196 7805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:44.872248 7806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:44.872269 7808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:44.873329 7800 server_base.cc:1034] running on GCE node
I20260925 01:21:44.874015 7800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:44.874984 7800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:44.876242 7800 hybrid_clock.cc:648] HybridClock initialized: now 1790299304876193 us; error 43 us; skew 500 ppm
I20260925 01:21:44.876602 7800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:44 dist-test-slave-s12s krb5kdc[7694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299304, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:44.879956 7800 init.cc:377] Logged in from keytab as kudu/127.20.102.129@KRBTEST.COM (short username kudu)
I20260925 01:21:44.881494 7800 webserver.cc:460] Webserver started at http://127.20.102.129:41349/ using document root <none> and password file <none>
I20260925 01:21:44.882153 7800 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:44.882256 7800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:44.882517 7800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:44.885459 7800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "fb0924fe299148879ba0e6492215c0a5"
format_stamp: "Formatted at 2026-09-25 01:21:44 on dist-test-slave-s12s"
server_key: "c0d7ec32d22de3f2c9c276bdb517acc5"
server_key_iv: "fbf6b030433f69af72e9ce77a4a1d878"
server_key_version: "encryptionkey@0"
I20260925 01:21:44.886219 7800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "fb0924fe299148879ba0e6492215c0a5"
format_stamp: "Formatted at 2026-09-25 01:21:44 on dist-test-slave-s12s"
server_key: "c0d7ec32d22de3f2c9c276bdb517acc5"
server_key_iv: "fbf6b030433f69af72e9ce77a4a1d878"
server_key_version: "encryptionkey@0"
I20260925 01:21:44.891424 7800 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:21:44.894869 7815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:44.896204 7800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:21:44.896339 7800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "fb0924fe299148879ba0e6492215c0a5"
format_stamp: "Formatted at 2026-09-25 01:21:44 on dist-test-slave-s12s"
server_key: "c0d7ec32d22de3f2c9c276bdb517acc5"
server_key_iv: "fbf6b030433f69af72e9ce77a4a1d878"
server_key_version: "encryptionkey@0"
I20260925 01:21:44.896525 7800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:44.907702 7800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:44.911787 7800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:44.912070 7800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:44.913164 7800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:44.914688 7800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:44.914769 7800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:44.914865 7800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:44.914919 7800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:44.956190 7800 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:36015
I20260925 01:21:44.956238 7927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:36015 every 8 connection(s)
I20260925 01:21:44.957688 7800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:21:44.964490 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 7800
I20260925 01:21:44.964731 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:21:44.965303 20890 external_mini_cluster.cc:1428] Setting key eafdc618f807c9d8e3e85c979f3d86ef
Sep 25 01:21:44 dist-test-slave-s12s krb5kdc[7694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299304, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.129@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:21:44.981837 7930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:44.960546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:42707 (local address 127.20.102.190:35351)
0925 01:21:44.965103 (+ 4557us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:44.965109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:44.965131 (+ 22us) server_negotiation.cc:407] Connection header received
0925 01:21:44.965200 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:44.965205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:44.965286 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:44.965415 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:44.967457 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:44.968550 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:44.970375 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:44.971094 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:44.971245 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:44.975143 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:44.975153 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:44.975160 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:44.975209 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:44.978096 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:44.979190 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:44.979194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:44.979197 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:44.979318 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:44.979891 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:44.979894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:44.979896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:44.980168 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:44.980288 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:44.981365 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0925 01:21:44.981560 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4408,"thread_start_us":4289,"threads_started":1}
I20260925 01:21:44.983573 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:44.961216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35351 (local address 127.20.102.129:42707)
0925 01:21:44.962180 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:44.962231 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:44.963909 (+ 1678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:44.965772 (+ 1863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:44.965781 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:44.966473 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:44.967232 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:44.967254 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:44.968799 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:44.968804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:44.970207 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:44.970221 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:44.971340 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:44.971346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:44.971581 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:44.972372 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:21:44.972399 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:44.974894 (+ 2495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:44.978266 (+ 3372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:44.978279 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:44.978301 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:44.979054 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:44.979436 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:44.979444 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:44.979450 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:44.979760 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:44.980317 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:44.980329 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:44.980906 (+ 577us) client_negotiation.cc:769] Sending connection context
0925 01:21:44.981247 (+ 341us) client_negotiation.cc:240] Negotiation successful
0925 01:21:44.981913 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":422,"threads_started":1}
I20260925 01:21:44.985358 7928 heartbeater.cc:347] Connected to a master server at 127.20.102.190:35351
I20260925 01:21:44.985740 7928 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:44.986505 7928 heartbeater.cc:511] Master 127.20.102.190:35351 requested a full tablet report, sending...
I20260925 01:21:44.988919 7739 ts_manager.cc:194] Registered new tserver with Master: fb0924fe299148879ba0e6492215c0a5 (127.20.102.129:36015)
I20260925 01:21:44.990465 7739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.129@KRBTEST.COM'} at 127.20.102.129:42707
WARNING: no policy specified for kudu/127.20.102.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
Entry for principal kudu/127.20.102.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab.
I20260925 01:21:45.025888 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35351
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:45.355762 7934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:45.356040 7934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:45.356204 7934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:45.363690 7934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:45.363817 7934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:45.364009 7934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:21:45.373870 7934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.130.keytab
--principal=kudu/127.20.102.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:45.375267 7934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:45.376799 7934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:45.389982 7939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:45.389991 7942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:45.390336 7940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:45.390738 7934 server_base.cc:1034] running on GCE node
I20260925 01:21:45.391399 7934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:45.392310 7934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:45.393512 7934 hybrid_clock.cc:648] HybridClock initialized: now 1790299305393451 us; error 50 us; skew 500 ppm
I20260925 01:21:45.393846 7934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:45 dist-test-slave-s12s krb5kdc[7694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299305, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:45.397148 7934 init.cc:377] Logged in from keytab as kudu/127.20.102.130@KRBTEST.COM (short username kudu)
I20260925 01:21:45.398644 7934 webserver.cc:460] Webserver started at http://127.20.102.130:44617/ using document root <none> and password file <none>
I20260925 01:21:45.399309 7934 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:45.399410 7934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:45.399683 7934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:45.402829 7934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "56f8bf70e5db43dc9593772e921a1567"
format_stamp: "Formatted at 2026-09-25 01:21:45 on dist-test-slave-s12s"
server_key: "f914c300aee7cb36267f2fb0e32e08cd"
server_key_iv: "0f559170dd6c2cc0fd0829be6252bc28"
server_key_version: "encryptionkey@0"
I20260925 01:21:45.403570 7934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "56f8bf70e5db43dc9593772e921a1567"
format_stamp: "Formatted at 2026-09-25 01:21:45 on dist-test-slave-s12s"
server_key: "f914c300aee7cb36267f2fb0e32e08cd"
server_key_iv: "0f559170dd6c2cc0fd0829be6252bc28"
server_key_version: "encryptionkey@0"
I20260925 01:21:45.408883 7934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 01:21:45.412365 7949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:45.414078 7934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:21:45.414273 7934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "56f8bf70e5db43dc9593772e921a1567"
format_stamp: "Formatted at 2026-09-25 01:21:45 on dist-test-slave-s12s"
server_key: "f914c300aee7cb36267f2fb0e32e08cd"
server_key_iv: "0f559170dd6c2cc0fd0829be6252bc28"
server_key_version: "encryptionkey@0"
I20260925 01:21:45.414474 7934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:45.437227 7934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:45.441887 7934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:45.442198 7934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:45.443269 7934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:45.444550 7934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:45.444621 7934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:45.444789 7934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:45.444860 7934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:45.479296 7934 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:38377
I20260925 01:21:45.479382 8061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:38377 every 8 connection(s)
I20260925 01:21:45.480667 7934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:21:45.490732 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 7934
I20260925 01:21:45.490945 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:21:45.491564 20890 external_mini_cluster.cc:1428] Setting key d33ee92a84cde11c0c55059ac90422e7
Sep 25 01:21:45 dist-test-slave-s12s krb5kdc[7694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299305, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.130@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:21:45.505101 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:45.483358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:52035 (local address 127.20.102.190:35351)
0925 01:21:45.483844 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:45.483850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:45.484962 (+ 1112us) server_negotiation.cc:407] Connection header received
0925 01:21:45.487322 (+ 2360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:45.487327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:45.487396 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:45.487491 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:45.489036 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:45.489792 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:45.492310 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:45.492920 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:45.493023 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:45.498052 (+ 5029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:45.498058 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:45.498061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:45.498098 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:45.501226 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:45.502260 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:45.502264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:45.502267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:45.502427 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:45.502950 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:45.502953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:45.502955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:45.503228 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:45.503340 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:45.504345 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0925 01:21:45.504598 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":230,"threads_started":1}
I20260925 01:21:45.506043 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:45.483990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35351 (local address 127.20.102.130:52035)
0925 01:21:45.484800 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:45.484847 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:45.487074 (+ 2227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:45.487692 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:45.487699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:45.488200 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:45.488857 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:45.488877 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:45.490017 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:45.490021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:45.492135 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:45.492152 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:45.493161 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:45.493170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:45.493478 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:45.494318 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:21:45.494344 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:45.497764 (+ 3420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:45.501416 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:45.501428 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:45.501449 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:45.502122 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:45.502559 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:45.502566 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:45.502572 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:45.502835 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:45.503368 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:45.503390 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:45.503897 (+ 507us) client_negotiation.cc:769] Sending connection context
0925 01:21:45.504264 (+ 367us) client_negotiation.cc:240] Negotiation successful
0925 01:21:45.504854 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":261,"threads_started":1}
I20260925 01:21:45.507779 8062 heartbeater.cc:347] Connected to a master server at 127.20.102.190:35351
I20260925 01:21:45.508144 8062 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:45.509074 8062 heartbeater.cc:511] Master 127.20.102.190:35351 requested a full tablet report, sending...
I20260925 01:21:45.510860 7739 ts_manager.cc:194] Registered new tserver with Master: 56f8bf70e5db43dc9593772e921a1567 (127.20.102.130:38377)
I20260925 01:21:45.511732 7739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.130@KRBTEST.COM'} at 127.20.102.130:52035
WARNING: no policy specified for kudu/127.20.102.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
Entry for principal kudu/127.20.102.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab.
I20260925 01:21:45.554193 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35351
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:45.887602 8068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:45.887905 8068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:45.888000 8068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:45.895331 8068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:45.895444 8068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:45.895560 8068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:21:45.905359 8068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.131.keytab
--principal=kudu/127.20.102.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:35351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:45.906613 8068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:45.907881 8068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:45.921353 8073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:45.921519 8076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:45.921974 8074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:45.922266 8068 server_base.cc:1034] running on GCE node
I20260925 01:21:45.922923 8068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:45.923913 8068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:45.925148 8068 hybrid_clock.cc:648] HybridClock initialized: now 1790299305925100 us; error 56 us; skew 500 ppm
I20260925 01:21:45.925475 8068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:45 dist-test-slave-s12s krb5kdc[7694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299305, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:45.928579 8068 init.cc:377] Logged in from keytab as kudu/127.20.102.131@KRBTEST.COM (short username kudu)
I20260925 01:21:45.930092 8068 webserver.cc:460] Webserver started at http://127.20.102.131:43415/ using document root <none> and password file <none>
I20260925 01:21:45.930783 8068 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:45.930932 8068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:45.931257 8068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:45.934231 8068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "bca54374319245fe843c4c11e450d454"
format_stamp: "Formatted at 2026-09-25 01:21:45 on dist-test-slave-s12s"
server_key: "c66869c4a134e66819a8960c1d57580e"
server_key_iv: "53522770146189ba23f456be406d9232"
server_key_version: "encryptionkey@0"
I20260925 01:21:45.934988 8068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "bca54374319245fe843c4c11e450d454"
format_stamp: "Formatted at 2026-09-25 01:21:45 on dist-test-slave-s12s"
server_key: "c66869c4a134e66819a8960c1d57580e"
server_key_iv: "53522770146189ba23f456be406d9232"
server_key_version: "encryptionkey@0"
I20260925 01:21:45.940073 8068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:21:45.943656 8083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:45.945087 8068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:21:45.945221 8068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "bca54374319245fe843c4c11e450d454"
format_stamp: "Formatted at 2026-09-25 01:21:45 on dist-test-slave-s12s"
server_key: "c66869c4a134e66819a8960c1d57580e"
server_key_iv: "53522770146189ba23f456be406d9232"
server_key_version: "encryptionkey@0"
I20260925 01:21:45.945395 8068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:45.958226 8068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:45.963307 8068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:45.963541 8068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:45.964450 8068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:45.965838 8068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:45.965914 8068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:45.965998 8068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:45.966042 8068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:45.993942 7928 heartbeater.cc:503] Master 127.20.102.190:35351 was elected leader, sending a full tablet report...
I20260925 01:21:45.999514 8068 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:40519
I20260925 01:21:45.999620 8195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:40519 every 8 connection(s)
I20260925 01:21:46.000976 8068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:21:46.010162 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 8068
I20260925 01:21:46.010351 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:21:46.010850 20890 external_mini_cluster.cc:1428] Setting key ec4243ee8b1ecc423382bc26377d7224
Sep 25 01:21:46 dist-test-slave-s12s krb5kdc[7694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299305, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.131@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:21:46.021749 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.003836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:33565 (local address 127.20.102.190:35351)
0925 01:21:46.004011 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:46.004015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:46.005308 (+ 1293us) server_negotiation.cc:407] Connection header received
0925 01:21:46.007150 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:46.007156 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:46.007267 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:46.007377 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:46.008953 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.009707 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.011464 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.011990 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.012131 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.015918 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:46.015923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:46.015926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:46.015959 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:46.018850 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:46.019645 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:46.019649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:46.019651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:46.019792 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:46.020179 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:46.020182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:46.020184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:46.020470 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:46.020638 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:46.021315 (+ 677us) server_negotiation.cc:299] Negotiation successful
0925 01:21:46.021499 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:21:46.022703 8198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.004390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:35351 (local address 127.20.102.131:33565)
0925 01:21:46.005139 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:46.005187 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:46.006883 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:46.007615 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:46.007623 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:46.008126 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:46.008784 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:46.008804 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.009865 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.009878 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:46.011306 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:46.011323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.012342 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.012347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:46.012616 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.013400 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:21:46.013422 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:46.015736 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:46.018992 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:46.018999 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:46.019012 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:46.019463 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:46.019890 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:21:46.019893 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:21:46.019895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:21:46.020073 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:21:46.020573 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:46.020579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:46.020947 (+ 368us) client_negotiation.cc:769] Sending connection context
0925 01:21:46.021190 (+ 243us) client_negotiation.cc:240] Negotiation successful
0925 01:21:46.021578 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":334,"threads_started":1}
I20260925 01:21:46.024179 8196 heartbeater.cc:347] Connected to a master server at 127.20.102.190:35351
I20260925 01:21:46.024492 8196 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:46.025352 8196 heartbeater.cc:511] Master 127.20.102.190:35351 requested a full tablet report, sending...
I20260925 01:21:46.026737 7739 ts_manager.cc:194] Registered new tserver with Master: bca54374319245fe843c4c11e450d454 (127.20.102.131:40519)
I20260925 01:21:46.027591 7739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.20.102.131@KRBTEST.COM'} at 127.20.102.131:33565
I20260925 01:21:46.038590 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:21:46.056064 8064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.042633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.20.102.190:35351)
0925 01:21:46.042955 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:46.042964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:46.043371 (+ 407us) server_negotiation.cc:407] Connection header received
0925 01:21:46.043573 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:46.043580 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:46.043668 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:46.043775 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:46.045305 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.046154 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.047413 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.047857 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.047996 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.050591 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:46.050597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:46.050602 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:46.050647 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:46.053232 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:46.054023 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:46.054030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:46.054034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:46.054182 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:46.054572 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:46.054578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:46.054582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:46.054898 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:46.055179 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:46.055506 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 01:21:46.055736 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:21:46.059966 7739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:21:46.067355 7739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:21:46.100360 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.082961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53592 (local address 127.20.102.129:36015)
0925 01:21:46.091583 (+ 8622us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:46.091596 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:46.091634 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:21:46.091762 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:46.091773 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:46.092116 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:46.092430 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:46.093252 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.094514 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.097431 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.098984 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.099193 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.099386 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:46.099780 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 01:21:46.099996 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8371,"thread_start_us":8181,"threads_started":1}
I20260925 01:21:46.100419 8209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.082771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:36015 (local address 127.0.0.1:53592)
0925 01:21:46.090646 (+ 7875us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:46.090669 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:46.090810 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:46.092468 (+ 1658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:46.092480 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:46.092525 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:46.093025 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:46.093037 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.094672 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.094677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:46.097263 (+ 2586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:46.097283 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.099227 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.099234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:46.099494 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.099559 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 01:21:46.099698 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 01:21:46.099962 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7701,"mutex_wait_us":1,"thread_start_us":7104,"threads_started":1}
I20260925 01:21:46.100924 8210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.080714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48486 (local address 127.20.102.131:40519)
0925 01:21:46.093318 (+ 12604us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:46.093329 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:46.093369 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:21:46.093489 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:46.093499 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:46.093797 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:46.094029 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:46.094649 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.096403 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.098517 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.099693 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.099912 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.100069 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:46.100382 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 01:21:46.100557 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12377,"thread_start_us":12167,"threads_started":1}
I20260925 01:21:46.101296 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.080206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:40519 (local address 127.0.0.1:48486)
0925 01:21:46.081221 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:46.081281 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:46.081486 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:46.094031 (+ 12545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:46.094036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:46.094062 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:46.094431 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:46.094446 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.096559 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.096570 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:46.098352 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:46.098364 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.099956 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.099962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:46.100154 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.100189 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 01:21:46.100306 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 01:21:46.100459 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":1,"thread_start_us":820,"threads_started":1}
I20260925 01:21:46.103421 8212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.080394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:38377 (local address 127.0.0.1:46256)
0925 01:21:46.090201 (+ 9807us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:46.090229 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:46.090398 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:46.095165 (+ 4767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:46.095170 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:46.095194 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:46.095718 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:46.095728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.097528 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.097533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:46.099604 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:46.099616 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.102661 (+ 3045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.102668 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:46.102865 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.102913 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 01:21:46.103006 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 01:21:46.103165 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9636,"mutex_wait_us":72,"thread_start_us":10122,"threads_started":1}
I20260925 01:21:46.104035 8211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.080860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46256 (local address 127.20.102.130:38377)
0925 01:21:46.094733 (+ 13873us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:46.094739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:46.094760 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:21:46.094829 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:46.094834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:46.094979 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:46.095177 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:46.095896 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.097330 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.100611 (+ 3281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.102174 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.102368 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.102595 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:46.103652 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0925 01:21:46.103797 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13725,"thread_start_us":13582,"threads_started":1}
I20260925 01:21:46.105250 7863 tablet_service.cc:1467] Processing CreateTablet for tablet f16bf967ab5144ceaf9cf171f3bd87cd (DEFAULT_TABLE table=test-table [id=dc31449c64ea4cd98c1b7f35fe5aec5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:46.106563 7863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16bf967ab5144ceaf9cf171f3bd87cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:46.107008 7997 tablet_service.cc:1467] Processing CreateTablet for tablet f16bf967ab5144ceaf9cf171f3bd87cd (DEFAULT_TABLE table=test-table [id=dc31449c64ea4cd98c1b7f35fe5aec5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:46.108423 7997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16bf967ab5144ceaf9cf171f3bd87cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:46.109472 8131 tablet_service.cc:1467] Processing CreateTablet for tablet f16bf967ab5144ceaf9cf171f3bd87cd (DEFAULT_TABLE table=test-table [id=dc31449c64ea4cd98c1b7f35fe5aec5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:46.110937 8131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f16bf967ab5144ceaf9cf171f3bd87cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:46.120249 8214 tablet_bootstrap.cc:492] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5: Bootstrap starting.
I20260925 01:21:46.123111 8215 tablet_bootstrap.cc:492] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567: Bootstrap starting.
I20260925 01:21:46.123934 8214 tablet_bootstrap.cc:654] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:46.125294 8214 log.cc:824] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:46.125610 8216 tablet_bootstrap.cc:492] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454: Bootstrap starting.
I20260925 01:21:46.127025 8215 tablet_bootstrap.cc:654] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:46.128345 8215 log.cc:824] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:46.128636 8216 tablet_bootstrap.cc:654] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:46.129384 8214 tablet_bootstrap.cc:492] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5: No bootstrap required, opened a new log
I20260925 01:21:46.129935 8216 log.cc:824] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:46.130158 8214 ts_tablet_manager.cc:1397] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260925 01:21:46.131567 8215 tablet_bootstrap.cc:492] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567: No bootstrap required, opened a new log
I20260925 01:21:46.131839 8215 ts_tablet_manager.cc:1397] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 01:21:46.132644 8216 tablet_bootstrap.cc:492] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454: No bootstrap required, opened a new log
I20260925 01:21:46.132943 8216 ts_tablet_manager.cc:1397] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260925 01:21:46.139690 8216 raft_consensus.cc:348] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca54374319245fe843c4c11e450d454" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 40519 } } peers { permanent_uuid: "56f8bf70e5db43dc9593772e921a1567" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38377 } } peers { permanent_uuid: "fb0924fe299148879ba0e6492215c0a5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36015 } }
I20260925 01:21:46.140127 8216 raft_consensus.cc:374] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:46.140264 8216 raft_consensus.cc:729] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bca54374319245fe843c4c11e450d454, State: Initialized, Role: FOLLOWER
I20260925 01:21:46.140995 8216 consensus_queue.cc:260] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca54374319245fe843c4c11e450d454" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 40519 } } peers { permanent_uuid: "56f8bf70e5db43dc9593772e921a1567" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38377 } } peers { permanent_uuid: "fb0924fe299148879ba0e6492215c0a5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36015 } }
I20260925 01:21:46.141289 8214 raft_consensus.cc:348] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca54374319245fe843c4c11e450d454" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 40519 } } peers { permanent_uuid: "56f8bf70e5db43dc9593772e921a1567" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38377 } } peers { permanent_uuid: "fb0924fe299148879ba0e6492215c0a5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36015 } }
I20260925 01:21:46.141897 8214 raft_consensus.cc:374] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:46.142047 8214 raft_consensus.cc:729] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb0924fe299148879ba0e6492215c0a5, State: Initialized, Role: FOLLOWER
I20260925 01:21:46.142180 8196 heartbeater.cc:503] Master 127.20.102.190:35351 was elected leader, sending a full tablet report...
I20260925 01:21:46.142956 8216 ts_tablet_manager.cc:1428] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260925 01:21:46.143283 8214 consensus_queue.cc:260] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca54374319245fe843c4c11e450d454" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 40519 } } peers { permanent_uuid: "56f8bf70e5db43dc9593772e921a1567" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38377 } } peers { permanent_uuid: "fb0924fe299148879ba0e6492215c0a5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36015 } }
I20260925 01:21:46.144275 8215 raft_consensus.cc:348] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca54374319245fe843c4c11e450d454" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 40519 } } peers { permanent_uuid: "56f8bf70e5db43dc9593772e921a1567" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38377 } } peers { permanent_uuid: "fb0924fe299148879ba0e6492215c0a5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36015 } }
I20260925 01:21:46.144734 8215 raft_consensus.cc:374] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:46.144840 8215 raft_consensus.cc:729] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56f8bf70e5db43dc9593772e921a1567, State: Initialized, Role: FOLLOWER
I20260925 01:21:46.145468 8215 consensus_queue.cc:260] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca54374319245fe843c4c11e450d454" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 40519 } } peers { permanent_uuid: "56f8bf70e5db43dc9593772e921a1567" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38377 } } peers { permanent_uuid: "fb0924fe299148879ba0e6492215c0a5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36015 } }
I20260925 01:21:46.146872 8062 heartbeater.cc:503] Master 127.20.102.190:35351 was elected leader, sending a full tablet report...
I20260925 01:21:46.147322 8215 ts_tablet_manager.cc:1428] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260925 01:21:46.149225 8214 ts_tablet_manager.cc:1428] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5: Time spent starting tablet: real 0.019s user 0.013s sys 0.000s
I20260925 01:21:46.190831 8221 raft_consensus.cc:482] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:21:46.191200 8221 raft_consensus.cc:504] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca54374319245fe843c4c11e450d454" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 40519 } } peers { permanent_uuid: "56f8bf70e5db43dc9593772e921a1567" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38377 } } peers { permanent_uuid: "fb0924fe299148879ba0e6492215c0a5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36015 } }
I20260925 01:21:46.193162 8221 leader_election.cc:288] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bca54374319245fe843c4c11e450d454 (127.20.102.131:40519), 56f8bf70e5db43dc9593772e921a1567 (127.20.102.130:38377)
I20260925 01:21:46.200603 8210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.193543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:39117 (local address 127.20.102.131:40519)
0925 01:21:46.193792 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:46.193801 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:46.194824 (+ 1023us) server_negotiation.cc:407] Connection header received
0925 01:21:46.194987 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:46.195009 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:46.195089 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:46.195182 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:46.196017 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.196819 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.198729 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.199489 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.199693 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.199776 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:46.200167 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 01:21:46.200315 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 01:21:46.200726 8211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.193920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:40587 (local address 127.20.102.130:38377)
0925 01:21:46.194186 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:46.194195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:46.194636 (+ 441us) server_negotiation.cc:407] Connection header received
0925 01:21:46.194829 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:46.194836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:46.194930 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:46.195027 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:46.195832 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.196725 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.198613 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.199536 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.199747 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.199849 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:46.200221 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 01:21:46.200375 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 01:21:46.200780 8223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.193545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:40519 (local address 127.20.102.129:39117)
0925 01:21:46.194653 (+ 1108us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:46.194674 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:46.194848 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:46.195225 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:46.195228 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:46.195284 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:46.195837 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:46.195851 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.196979 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.196983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:46.198572 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:46.198590 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.199740 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.199747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:46.199917 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.199941 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:21:46.200138 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 01:21:46.200337 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":43,"thread_start_us":647,"threads_started":1}
I20260925 01:21:46.200978 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.193483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:38377 (local address 127.20.102.129:40587)
0925 01:21:46.194528 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:46.194545 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:46.194674 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:46.195066 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:46.195071 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:46.195096 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:46.195523 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:46.195534 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.196882 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.196890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:46.198444 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:46.198456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.199813 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.199820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:46.200042 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.200081 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 01:21:46.200191 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 01:21:46.200404 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":76,"thread_start_us":746,"threads_started":1}
I20260925 01:21:46.202083 8151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16bf967ab5144ceaf9cf171f3bd87cd" candidate_uuid: "fb0924fe299148879ba0e6492215c0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bca54374319245fe843c4c11e450d454" is_pre_election: true
I20260925 01:21:46.202135 8017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16bf967ab5144ceaf9cf171f3bd87cd" candidate_uuid: "fb0924fe299148879ba0e6492215c0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56f8bf70e5db43dc9593772e921a1567" is_pre_election: true
I20260925 01:21:46.202713 8151 raft_consensus.cc:2454] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb0924fe299148879ba0e6492215c0a5 in term 0.
I20260925 01:21:46.202791 8017 raft_consensus.cc:2454] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb0924fe299148879ba0e6492215c0a5 in term 0.
I20260925 01:21:46.203712 7816 leader_election.cc:302] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bca54374319245fe843c4c11e450d454, fb0924fe299148879ba0e6492215c0a5; no voters:
I20260925 01:21:46.204205 8221 raft_consensus.cc:2781] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:21:46.204381 8221 raft_consensus.cc:482] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:21:46.204497 8221 raft_consensus.cc:3037] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:46.206708 8221 raft_consensus.cc:504] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca54374319245fe843c4c11e450d454" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 40519 } } peers { permanent_uuid: "56f8bf70e5db43dc9593772e921a1567" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38377 } } peers { permanent_uuid: "fb0924fe299148879ba0e6492215c0a5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36015 } }
I20260925 01:21:46.207265 8221 leader_election.cc:288] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [CANDIDATE]: Term 1 election: Requested vote from peers bca54374319245fe843c4c11e450d454 (127.20.102.131:40519), 56f8bf70e5db43dc9593772e921a1567 (127.20.102.130:38377)
I20260925 01:21:46.207800 8151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16bf967ab5144ceaf9cf171f3bd87cd" candidate_uuid: "fb0924fe299148879ba0e6492215c0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bca54374319245fe843c4c11e450d454"
I20260925 01:21:46.207808 8017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f16bf967ab5144ceaf9cf171f3bd87cd" candidate_uuid: "fb0924fe299148879ba0e6492215c0a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56f8bf70e5db43dc9593772e921a1567"
I20260925 01:21:46.208006 8151 raft_consensus.cc:3037] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:46.208026 8017 raft_consensus.cc:3037] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567 [term 0 FOLLOWER]: Advancing to term 1
W20260925 01:21:46.210240 7929 tablet.cc:2369] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:21:46.210317 8151 raft_consensus.cc:2454] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb0924fe299148879ba0e6492215c0a5 in term 1.
I20260925 01:21:46.210567 8017 raft_consensus.cc:2454] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb0924fe299148879ba0e6492215c0a5 in term 1.
I20260925 01:21:46.210734 7816 leader_election.cc:302] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bca54374319245fe843c4c11e450d454, fb0924fe299148879ba0e6492215c0a5; no voters:
I20260925 01:21:46.211097 8221 raft_consensus.cc:2781] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:46.211585 8221 raft_consensus.cc:686] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [term 1 LEADER]: Becoming Leader. State: Replica: fb0924fe299148879ba0e6492215c0a5, State: Running, Role: LEADER
I20260925 01:21:46.212095 8221 consensus_queue.cc:237] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca54374319245fe843c4c11e450d454" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 40519 } } peers { permanent_uuid: "56f8bf70e5db43dc9593772e921a1567" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38377 } } peers { permanent_uuid: "fb0924fe299148879ba0e6492215c0a5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36015 } }
I20260925 01:21:46.216922 7738 catalog_manager.cc:5495] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb0924fe299148879ba0e6492215c0a5 (127.20.102.129). New cstate: current_term: 1 leader_uuid: "fb0924fe299148879ba0e6492215c0a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bca54374319245fe843c4c11e450d454" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 40519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "56f8bf70e5db43dc9593772e921a1567" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb0924fe299148879ba0e6492215c0a5" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 36015 } health_report { overall_health: HEALTHY } } }
W20260925 01:21:46.232926 8063 tablet.cc:2369] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:21:46.246142 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.240456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.20.102.129:36015)
0925 01:21:46.240772 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:46.240778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:46.240894 (+ 116us) server_negotiation.cc:407] Connection header received
0925 01:21:46.241147 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:46.241151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:46.241257 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:46.241415 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:21:46.241781 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.242509 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.243920 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.244325 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.244463 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.245047 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:21:46.245561 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:21:46.245735 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:46.245795 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 01:21:46.245878 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260925 01:21:46.252961 8197 tablet.cc:2369] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:21:46.254753 8151 raft_consensus.cc:1261] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454 [term 1 FOLLOWER]: Refusing update from remote peer fb0924fe299148879ba0e6492215c0a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:21:46.254755 8017 raft_consensus.cc:1261] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567 [term 1 FOLLOWER]: Refusing update from remote peer fb0924fe299148879ba0e6492215c0a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:21:46.255769 8225 consensus_queue.cc:1035] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bca54374319245fe843c4c11e450d454" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 40519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:21:46.256119 8221 consensus_queue.cc:1035] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56f8bf70e5db43dc9593772e921a1567" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 38377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:21:46.268772 8228 mvcc.cc:205] Tried to move back new op lower bound from 7333065958408404992 to 7333065958248775680. Current Snapshot: MvccSnapshot[applied={T|T < 7333065958408404992}]
I20260925 01:21:46.283560 7739 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:21:46.283885 7739 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:21:46.287746 7739 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bcbb2bb6d2b74802833d4162e7d1b40f: Sending DeleteTablet for 3 replicas of tablet f16bf967ab5144ceaf9cf171f3bd87cd
I20260925 01:21:46.289021 7997 tablet_service.cc:1514] Processing DeleteTablet for tablet f16bf967ab5144ceaf9cf171f3bd87cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:46 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:46256
I20260925 01:21:46.289021 8131 tablet_service.cc:1514] Processing DeleteTablet for tablet f16bf967ab5144ceaf9cf171f3bd87cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:46 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:48486
I20260925 01:21:46.289098 7863 tablet_service.cc:1514] Processing DeleteTablet for tablet f16bf967ab5144ceaf9cf171f3bd87cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:46 UTC) from {username='kudu', principal='kudu/127.20.102.190@KRBTEST.COM'} at 127.0.0.1:53592
I20260925 01:21:46.294373 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 7800
I20260925 01:21:46.301349 8238 tablet_replica.cc:330] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567: stopping tablet replica
I20260925 01:21:46.301545 8239 tablet_replica.cc:330] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5: stopping tablet replica
I20260925 01:21:46.301854 8238 raft_consensus.cc:2229] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:21:46.301982 8239 raft_consensus.cc:2229] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:21:46.302862 8239 raft_consensus.cc:2258] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:46.303949 8238 raft_consensus.cc:2258] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:46.305153 8237 tablet_replica.cc:330] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454: stopping tablet replica
I20260925 01:21:46.306469 8238 ts_tablet_manager.cc:1905] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:46.307022 8237 raft_consensus.cc:2229] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:21:46.307256 8239 ts_tablet_manager.cc:1905] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:46.307610 8237 raft_consensus.cc:2258] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:46.309845 8237 ts_tablet_manager.cc:1905] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:46.314100 8238 ts_tablet_manager.cc:1918] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:46.314337 8238 log.cc:1196] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/f16bf967ab5144ceaf9cf171f3bd87cd
I20260925 01:21:46.315029 8238 ts_tablet_manager.cc:1939] T f16bf967ab5144ceaf9cf171f3bd87cd P 56f8bf70e5db43dc9593772e921a1567: Deleting consensus metadata
I20260925 01:21:46.317094 8237 ts_tablet_manager.cc:1918] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:46.317341 8237 log.cc:1196] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/f16bf967ab5144ceaf9cf171f3bd87cd
I20260925 01:21:46.317968 8237 ts_tablet_manager.cc:1939] T f16bf967ab5144ceaf9cf171f3bd87cd P bca54374319245fe843c4c11e450d454: Deleting consensus metadata
I20260925 01:21:46.320191 7723 catalog_manager.cc:4841] TS bca54374319245fe843c4c11e450d454 (127.20.102.131:40519): tablet f16bf967ab5144ceaf9cf171f3bd87cd (table test-table [id=dc31449c64ea4cd98c1b7f35fe5aec5d]) successfully deleted
I20260925 01:21:46.328964 8239 ts_tablet_manager.cc:1918] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:46.329151 8239 log.cc:1196] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790298989998230-20890-0/minicluster-data/ts-0/wal/wals/f16bf967ab5144ceaf9cf171f3bd87cd
I20260925 01:21:46.329926 8239 ts_tablet_manager.cc:1939] T f16bf967ab5144ceaf9cf171f3bd87cd P fb0924fe299148879ba0e6492215c0a5: Deleting consensus metadata
Sep 25 01:21:46 dist-test-slave-s12s krb5kdc[7694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.129@KRBTEST.COM
I20260925 01:21:46.335270 7725 catalog_manager.cc:4841] TS fb0924fe299148879ba0e6492215c0a5 (127.20.102.129:36015): tablet f16bf967ab5144ceaf9cf171f3bd87cd (table test-table [id=dc31449c64ea4cd98c1b7f35fe5aec5d]) successfully deleted
I20260925 01:21:46.344501 7726 catalog_manager.cc:4841] TS 56f8bf70e5db43dc9593772e921a1567 (127.20.102.130:38377): tablet f16bf967ab5144ceaf9cf171f3bd87cd (table test-table [id=dc31449c64ea4cd98c1b7f35fe5aec5d]) successfully deleted
I20260925 01:21:46.345606 8213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.295737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53606 (local address 127.20.102.129:36015)
0925 01:21:46.295913 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:46.295919 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:46.313927 (+ 18008us) server_negotiation.cc:407] Connection header received
0925 01:21:46.314479 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:46.314485 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:46.314577 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:46.314663 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:46.316520 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.317730 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.326383 (+ 8653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.327356 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.327466 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.336333 (+ 8867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:46.336350 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:46.336370 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:46.336420 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:46.340315 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:46.341259 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:46.341269 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:46.341273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:46.341463 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:46.341949 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:46.341956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:46.341961 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:46.342424 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:46.342718 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:46.344948 (+ 2230us) server_negotiation.cc:299] Negotiation successful
0925 01:21:46.345209 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:21:46.347220 7923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:46.453567 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 7800
I20260925 01:21:46.474170 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 7934
Sep 25 01:21:46 dist-test-slave-s12s krb5kdc[7694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.130@KRBTEST.COM
I20260925 01:21:46.491168 8211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.474917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.20.102.130:38377)
0925 01:21:46.475108 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:46.475113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:46.475131 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:21:46.475275 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:46.475278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:46.475343 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:46.475428 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:46.476958 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.477713 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.479244 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.479686 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.479787 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.485036 (+ 5249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:46.485047 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:46.485073 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:46.485110 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:46.488148 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:46.489108 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:46.489115 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:46.489118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:46.489238 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:46.489654 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:46.489658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:46.489660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:46.490077 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:46.490286 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:46.490659 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 01:21:46.490827 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 01:21:46.492004 8057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:46.582180 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 7934
I20260925 01:21:46.601143 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 8068
Sep 25 01:21:46 dist-test-slave-s12s krb5kdc[7694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.131@KRBTEST.COM
I20260925 01:21:46.619927 8210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:46.602022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.20.102.131:40519)
0925 01:21:46.602227 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:46.602232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:46.602252 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:21:46.602392 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:46.602396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:46.602464 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:46.602549 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:46.604522 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.605887 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.607679 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:46.608464 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:46.608623 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:46.614078 (+ 5455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:46.614095 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:46.614125 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:46.614168 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:46.617033 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:46.617720 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:46.617730 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:46.617734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:46.617892 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:46.618275 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:46.618281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:46.618285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:46.618646 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:46.618967 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:46.619341 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 01:21:46.619585 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 01:21:46.620580 8191 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:46.709712 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 8068
I20260925 01:21:46.728096 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 7707
I20260925 01:21:46.729408 7769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:46.844190 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 7707
2026-09-25T01:21:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4360 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T01:21:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:21:46Z Disabled control of system clock
I20260925 01:21:46.925176 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:36927
--webserver_interface=127.20.102.190
--webserver_port=0
--builtin_ntp_servers=127.20.102.148:45651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:36927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:21:47.269084 8248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:47.269392 8248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:47.269558 8248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:47.277216 8248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:21:47.277331 8248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:47.277419 8248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:47.277521 8248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:21:47.277608 8248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:21:47.287263 8248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:45651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:36927
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:47.288638 8248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:47.290627 8248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:47.302800 8256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:47.302873 8253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:47.302873 8254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:47.303769 8248 server_base.cc:1034] running on GCE node
I20260925 01:21:47.304510 8248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:47.306022 8248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:47.307240 8248 hybrid_clock.cc:648] HybridClock initialized: now 1790299307307198 us; error 29 us; skew 500 ppm
I20260925 01:21:47.307626 8248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:21:47.313911 8248 webserver.cc:460] Webserver started at http://127.20.102.190:33125/ using document root <none> and password file <none>
I20260925 01:21:47.314704 8248 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:47.314802 8248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:47.315155 8248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:47.318356 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "628a84eabefa4d609a99b05b9fabe731"
format_stamp: "Formatted at 2026-09-25 01:21:47 on dist-test-slave-s12s"
server_key: "217d311e2f06ba1a40d95d1323b48dab"
server_key_iv: "43beb42401c66c119cc1a9c0c137e450"
server_key_version: "encryptionkey@0"
I20260925 01:21:47.319231 8248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "628a84eabefa4d609a99b05b9fabe731"
format_stamp: "Formatted at 2026-09-25 01:21:47 on dist-test-slave-s12s"
server_key: "217d311e2f06ba1a40d95d1323b48dab"
server_key_iv: "43beb42401c66c119cc1a9c0c137e450"
server_key_version: "encryptionkey@0"
I20260925 01:21:47.324807 8248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 01:21:47.328409 8262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:47.329952 8248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 01:21:47.330165 8248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "628a84eabefa4d609a99b05b9fabe731"
format_stamp: "Formatted at 2026-09-25 01:21:47 on dist-test-slave-s12s"
server_key: "217d311e2f06ba1a40d95d1323b48dab"
server_key_iv: "43beb42401c66c119cc1a9c0c137e450"
server_key_version: "encryptionkey@0"
I20260925 01:21:47.330394 8248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:47.346784 8248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:47.347646 8248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:47.347949 8248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:47.366734 8248 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:36927
I20260925 01:21:47.366758 8313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:36927 every 8 connection(s)
I20260925 01:21:47.368079 8248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:21:47.370868 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 8248
I20260925 01:21:47.371057 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:21:47.371501 20890 external_mini_cluster.cc:1428] Setting key 0b571b34052c90306af37739099ea781
I20260925 01:21:47.372184 8314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:47.384920 8314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731: Bootstrap starting.
I20260925 01:21:47.385730 8317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:47.373473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.20.102.190:36927)
0925 01:21:47.374482 (+ 1009us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:47.374493 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:47.374534 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:21:47.376862 (+ 2328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:47.376888 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:47.377370 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:47.377775 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:47.378379 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:47.379663 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:47.381106 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:47.381796 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:47.382020 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:47.383223 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:47.383231 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:47.383244 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:47.383810 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:47.383884 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:47.383906 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:47.384035 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:47.384287 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 01:21:47.384588 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":366,"threads_started":1}
I20260925 01:21:47.388690 8314 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:47.389675 8314 log.cc:824] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:47.392577 8314 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731: No bootstrap required, opened a new log
I20260925 01:21:47.401336 8314 raft_consensus.cc:348] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628a84eabefa4d609a99b05b9fabe731" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36927 } }
I20260925 01:21:47.401810 8314 raft_consensus.cc:374] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:47.401939 8314 raft_consensus.cc:729] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628a84eabefa4d609a99b05b9fabe731, State: Initialized, Role: FOLLOWER
I20260925 01:21:47.402647 8314 consensus_queue.cc:260] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628a84eabefa4d609a99b05b9fabe731" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36927 } }
I20260925 01:21:47.402922 8314 raft_consensus.cc:388] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:21:47.403084 8314 raft_consensus.cc:482] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:21:47.403288 8314 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:47.405889 8314 raft_consensus.cc:504] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628a84eabefa4d609a99b05b9fabe731" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36927 } }
I20260925 01:21:47.406452 8314 leader_election.cc:302] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 628a84eabefa4d609a99b05b9fabe731; no voters:
I20260925 01:21:47.407248 8314 leader_election.cc:288] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:21:47.407419 8319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:47.408394 8319 raft_consensus.cc:686] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [term 1 LEADER]: Becoming Leader. State: Replica: 628a84eabefa4d609a99b05b9fabe731, State: Running, Role: LEADER
I20260925 01:21:47.409003 8319 consensus_queue.cc:237] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628a84eabefa4d609a99b05b9fabe731" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36927 } }
I20260925 01:21:47.409667 8314 sys_catalog.cc:564] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:21:47.411998 8320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "628a84eabefa4d609a99b05b9fabe731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628a84eabefa4d609a99b05b9fabe731" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36927 } } }
I20260925 01:21:47.412576 8320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:47.412118 8321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 628a84eabefa4d609a99b05b9fabe731. Latest consensus state: current_term: 1 leader_uuid: "628a84eabefa4d609a99b05b9fabe731" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "628a84eabefa4d609a99b05b9fabe731" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 36927 } } }
I20260925 01:21:47.412811 8321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:47.414613 8328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:21:47.420917 8328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:21:47.430200 8328 catalog_manager.cc:1347] Generated new cluster ID: bce9904b495e4d6887d0d5fa1ed31253
I20260925 01:21:47.430330 8328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:21:47.461079 8328 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:21:47.462991 8328 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:21:47.477056 8328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731: Generated new TSK 0
I20260925 01:21:47.478180 8328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:21:47.486117 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.129:0
--local_ip_for_outbound_sockets=127.20.102.129
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:36927
--builtin_ntp_servers=127.20.102.148:45651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 01:21:47.825838 8338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:47.826182 8338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:47.826357 8338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:47.833746 8338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:47.833874 8338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:47.834062 8338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.129
I20260925 01:21:47.843261 8338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:45651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.102.129
--webserver_port=0
--tserver_master_addrs=127.20.102.190:36927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.129
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:47.844619 8338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:47.846481 8338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:47.859781 8344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:47.859715 8346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:47.859820 8343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:47.860862 8338 server_base.cc:1034] running on GCE node
I20260925 01:21:47.861493 8338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:47.862452 8338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:47.863705 8338 hybrid_clock.cc:648] HybridClock initialized: now 1790299307863651 us; error 49 us; skew 500 ppm
I20260925 01:21:47.864039 8338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:21:47.866451 8338 webserver.cc:460] Webserver started at http://127.20.102.129:38599/ using document root <none> and password file <none>
I20260925 01:21:47.867218 8338 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:47.867360 8338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:47.867695 8338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:47.870718 8338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/data/instance:
uuid: "95f0a9cf9c264bbf876102dea0d5affb"
format_stamp: "Formatted at 2026-09-25 01:21:47 on dist-test-slave-s12s"
server_key: "a37a3a7b3798d32bdd619961bb7dbf61"
server_key_iv: "51aa6a5fc6a7eb8e0d23163128e2eff0"
server_key_version: "encryptionkey@0"
I20260925 01:21:47.871543 8338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance:
uuid: "95f0a9cf9c264bbf876102dea0d5affb"
format_stamp: "Formatted at 2026-09-25 01:21:47 on dist-test-slave-s12s"
server_key: "a37a3a7b3798d32bdd619961bb7dbf61"
server_key_iv: "51aa6a5fc6a7eb8e0d23163128e2eff0"
server_key_version: "encryptionkey@0"
I20260925 01:21:47.877110 8338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 01:21:47.880821 8352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:47.882148 8338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:21:47.882321 8338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/wal
uuid: "95f0a9cf9c264bbf876102dea0d5affb"
format_stamp: "Formatted at 2026-09-25 01:21:47 on dist-test-slave-s12s"
server_key: "a37a3a7b3798d32bdd619961bb7dbf61"
server_key_iv: "51aa6a5fc6a7eb8e0d23163128e2eff0"
server_key_version: "encryptionkey@0"
I20260925 01:21:47.882535 8338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:47.913944 8338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:47.914944 8338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:47.915268 8338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:47.916273 8338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:47.917758 8338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:47.917886 8338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:47.918025 8338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:47.918118 8338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:47.952915 8338 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.129:43149
I20260925 01:21:47.953003 8464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.129:43149 every 8 connection(s)
I20260925 01:21:47.954264 8338 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/data/info.pb
I20260925 01:21:47.955492 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 8338
I20260925 01:21:47.955677 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-0/wal/instance
I20260925 01:21:47.956266 20890 external_mini_cluster.cc:1428] Setting key 895010511db2f901f74bb34b9157954b
I20260925 01:21:47.960043 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.130:0
--local_ip_for_outbound_sockets=127.20.102.130
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:36927
--builtin_ntp_servers=127.20.102.148:45651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:21:47.970368 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:47.957174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:45967 (local address 127.20.102.190:36927)
0925 01:21:47.957650 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:47.957656 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:47.959210 (+ 1554us) server_negotiation.cc:407] Connection header received
0925 01:21:47.962144 (+ 2934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:47.962149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:47.962216 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:47.962310 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:47.963783 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:47.964506 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:47.967104 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:47.967499 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:47.967601 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:47.969363 (+ 1762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:47.969371 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:47.969378 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:47.969403 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:47.969457 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:47.969477 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:47.969577 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:47.969932 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 01:21:47.970049 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260925 01:21:47.971575 8468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:47.957928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:36927 (local address 127.20.102.129:45967)
0925 01:21:47.959028 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:47.959079 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:47.961819 (+ 2740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:47.962598 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:47.962612 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:47.962741 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:47.963583 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:47.963610 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:47.964664 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:47.964672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:47.966943 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:47.966961 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:47.967736 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:47.967745 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:47.968077 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:47.968971 (+ 894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:21:47.968976 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:47.969032 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:21:47.969147 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:47.969645 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:47.969661 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:47.969670 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 01:21:47.969869 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 01:21:47.970313 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":323,"threads_started":1}
I20260925 01:21:47.973327 8465 heartbeater.cc:347] Connected to a master server at 127.20.102.190:36927
I20260925 01:21:47.973655 8465 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:47.974435 8465 heartbeater.cc:511] Master 127.20.102.190:36927 requested a full tablet report, sending...
I20260925 01:21:47.976876 8279 ts_manager.cc:194] Registered new tserver with Master: 95f0a9cf9c264bbf876102dea0d5affb (127.20.102.129:43149)
I20260925 01:21:47.979163 8279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.102.129:45967
W20260925 01:21:48.297987 8469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:48.298305 8469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:48.298398 8469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:48.306133 8469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:48.306252 8469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:48.306363 8469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.130
I20260925 01:21:48.316071 8469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:45651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.102.130
--webserver_port=0
--tserver_master_addrs=127.20.102.190:36927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.130
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:48.317471 8469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:48.319342 8469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:48.333052 8475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:48.333103 8477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:48.333014 8474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:48.338361 8469 server_base.cc:1034] running on GCE node
I20260925 01:21:48.339056 8469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:48.340167 8469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:48.341384 8469 hybrid_clock.cc:648] HybridClock initialized: now 1790299308341313 us; error 58 us; skew 500 ppm
I20260925 01:21:48.341737 8469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:21:48.344053 8469 webserver.cc:460] Webserver started at http://127.20.102.130:41545/ using document root <none> and password file <none>
I20260925 01:21:48.344847 8469 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:48.344969 8469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:48.345316 8469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:48.348255 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/data/instance:
uuid: "b3dd999e8a0647ad94d654b086cbf5af"
format_stamp: "Formatted at 2026-09-25 01:21:48 on dist-test-slave-s12s"
server_key: "b9e5a54b3b06bacc3cad37a4536331a1"
server_key_iv: "677885a0fee2b3209ee8ec396e323f67"
server_key_version: "encryptionkey@0"
I20260925 01:21:48.349092 8469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance:
uuid: "b3dd999e8a0647ad94d654b086cbf5af"
format_stamp: "Formatted at 2026-09-25 01:21:48 on dist-test-slave-s12s"
server_key: "b9e5a54b3b06bacc3cad37a4536331a1"
server_key_iv: "677885a0fee2b3209ee8ec396e323f67"
server_key_version: "encryptionkey@0"
I20260925 01:21:48.354545 8469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:21:48.357959 8483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:48.359426 8469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:21:48.359647 8469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/wal
uuid: "b3dd999e8a0647ad94d654b086cbf5af"
format_stamp: "Formatted at 2026-09-25 01:21:48 on dist-test-slave-s12s"
server_key: "b9e5a54b3b06bacc3cad37a4536331a1"
server_key_iv: "677885a0fee2b3209ee8ec396e323f67"
server_key_version: "encryptionkey@0"
I20260925 01:21:48.359862 8469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:48.384426 8469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:48.385344 8469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:48.385667 8469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:48.386752 8469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:48.388334 8469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:48.388448 8469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:48.388582 8469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:48.388659 8469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:48.423696 8469 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.130:43983
I20260925 01:21:48.423934 8595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.130:43983 every 8 connection(s)
I20260925 01:21:48.425114 8469 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/data/info.pb
I20260925 01:21:48.431304 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 8469
I20260925 01:21:48.431538 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/wal/instance
I20260925 01:21:48.432104 20890 external_mini_cluster.cc:1428] Setting key 93cf8f61112c90e616871d8e79491b8b
I20260925 01:21:48.435303 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.102.131:0
--local_ip_for_outbound_sockets=127.20.102.131
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:36927
--builtin_ntp_servers=127.20.102.148:45651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 01:21:48.439502 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:48.427720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.130:40173 (local address 127.20.102.190:36927)
0925 01:21:48.427953 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:48.427960 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:48.429435 (+ 1475us) server_negotiation.cc:407] Connection header received
0925 01:21:48.431377 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:48.431382 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:48.431504 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:48.431658 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:48.433082 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.433917 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.436474 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.436952 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.437056 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:48.438735 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:48.438740 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:48.438743 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:48.438762 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:48.438795 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:48.438807 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:48.438884 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:48.439190 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 01:21:48.439275 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 01:21:48.441049 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:48.428318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:36927 (local address 127.20.102.130:40173)
0925 01:21:48.429272 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:48.429309 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:48.431028 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:48.431953 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:48.431966 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:48.432093 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:48.432902 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:48.432922 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.434069 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.434077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:48.436286 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:48.436306 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.437200 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.437210 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:48.437553 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:48.438339 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:21:48.438345 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:48.438400 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:21:48.438509 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:48.438915 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:48.438929 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:48.438939 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 01:21:48.439147 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 01:21:48.439588 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":333,"threads_started":1}
I20260925 01:21:48.442634 8596 heartbeater.cc:347] Connected to a master server at 127.20.102.190:36927
I20260925 01:21:48.442971 8596 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:48.443782 8596 heartbeater.cc:511] Master 127.20.102.190:36927 requested a full tablet report, sending...
I20260925 01:21:48.445425 8279 ts_manager.cc:194] Registered new tserver with Master: b3dd999e8a0647ad94d654b086cbf5af (127.20.102.130:43983)
I20260925 01:21:48.446290 8279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.102.130:40173
W20260925 01:21:48.770272 8599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:48.770569 8599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:48.770721 8599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:48.778264 8599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:48.778365 8599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:48.778533 8599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.102.131
I20260925 01:21:48.789227 8599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:45651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.102.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.102.131
--webserver_port=0
--tserver_master_addrs=127.20.102.190:36927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.102.131
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:48.790611 8599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:48.792456 8599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:48.805872 8604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:48.805969 8605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:48.805966 8607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:48.807190 8599 server_base.cc:1034] running on GCE node
I20260925 01:21:48.807835 8599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:48.808876 8599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:48.810096 8599 hybrid_clock.cc:648] HybridClock initialized: now 1790299308810058 us; error 29 us; skew 500 ppm
I20260925 01:21:48.810415 8599 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:21:48.812937 8599 webserver.cc:460] Webserver started at http://127.20.102.131:36333/ using document root <none> and password file <none>
I20260925 01:21:48.813752 8599 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:48.813890 8599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:48.814206 8599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:48.817296 8599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/data/instance:
uuid: "58a1a382f58b46bd8fefe05acfcdadea"
format_stamp: "Formatted at 2026-09-25 01:21:48 on dist-test-slave-s12s"
server_key: "f42027a473f5dd0ef658006726bdc74d"
server_key_iv: "505e8e25597e61511cb3a8f76574e60b"
server_key_version: "encryptionkey@0"
I20260925 01:21:48.818068 8599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance:
uuid: "58a1a382f58b46bd8fefe05acfcdadea"
format_stamp: "Formatted at 2026-09-25 01:21:48 on dist-test-slave-s12s"
server_key: "f42027a473f5dd0ef658006726bdc74d"
server_key_iv: "505e8e25597e61511cb3a8f76574e60b"
server_key_version: "encryptionkey@0"
I20260925 01:21:48.823338 8599 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:21:48.827013 8613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:48.828415 8599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:21:48.828611 8599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/wal
uuid: "58a1a382f58b46bd8fefe05acfcdadea"
format_stamp: "Formatted at 2026-09-25 01:21:48 on dist-test-slave-s12s"
server_key: "f42027a473f5dd0ef658006726bdc74d"
server_key_iv: "505e8e25597e61511cb3a8f76574e60b"
server_key_version: "encryptionkey@0"
I20260925 01:21:48.828902 8599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:48.846963 8599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:48.847862 8599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:48.848187 8599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:48.849236 8599 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 01:21:48.850771 8599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 01:21:48.850908 8599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:48.851037 8599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 01:21:48.851125 8599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:48.885404 8599 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.131:44757
I20260925 01:21:48.885422 8725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.131:44757 every 8 connection(s)
I20260925 01:21:48.886984 8599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/data/info.pb
I20260925 01:21:48.893999 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 8599
I20260925 01:21:48.894203 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/wal/instance
I20260925 01:21:48.894773 20890 external_mini_cluster.cc:1428] Setting key de0a0d8e59dff724dc722a4d0c97ed67
I20260925 01:21:48.900991 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:48.890060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.131:41221 (local address 127.20.102.190:36927)
0925 01:21:48.890273 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:48.890278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:48.891695 (+ 1417us) server_negotiation.cc:407] Connection header received
0925 01:21:48.893678 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:48.893684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:48.893762 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:48.893843 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:48.895402 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.896437 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.898340 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.898754 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.898850 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:48.900137 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:48.900146 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:48.900154 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:48.900180 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:48.900232 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:48.900249 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:48.900340 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:48.900536 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 01:21:48.900614 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 01:21:48.902165 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:48.890544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:36927 (local address 127.20.102.131:41221)
0925 01:21:48.891520 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:48.891566 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:48.893350 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:48.894158 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:48.894172 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:48.894304 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:21:48.895174 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:48.895200 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.896624 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.896630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:48.898208 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:48.898220 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.898939 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.898944 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:48.899154 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:48.899811 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 01:21:48.899813 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:21:48.899854 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 01:21:48.899929 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:21:48.900346 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:21:48.900354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:21:48.900360 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 01:21:48.900506 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 01:21:48.900979 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":290,"threads_started":1}
I20260925 01:21:48.903502 8726 heartbeater.cc:347] Connected to a master server at 127.20.102.190:36927
I20260925 01:21:48.903818 8726 heartbeater.cc:464] Registering TS with master...
I20260925 01:21:48.904580 8726 heartbeater.cc:511] Master 127.20.102.190:36927 requested a full tablet report, sending...
I20260925 01:21:48.905988 8279 ts_manager.cc:194] Registered new tserver with Master: 58a1a382f58b46bd8fefe05acfcdadea (127.20.102.131:44757)
I20260925 01:21:48.906770 8279 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.102.131:41221
I20260925 01:21:48.910152 20890 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 01:21:48.920508 8467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:48.914598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46976 (local address 127.20.102.190:36927)
0925 01:21:48.914850 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:48.914857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:48.915197 (+ 340us) server_negotiation.cc:407] Connection header received
0925 01:21:48.915298 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:48.915302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:48.915399 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:48.915497 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:48.915993 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.916817 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.918127 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.918560 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.918652 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:48.919788 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:48.919793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:48.919796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:48.919814 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:48.919849 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:48.919860 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:48.919936 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:48.920072 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 01:21:48.920153 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 01:21:48.924858 8279 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 01:21:48.932005 8279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 01:21:48.973186 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:48.945884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.20.102.131:44757)
0925 01:21:48.964262 (+ 18378us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:48.964273 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:48.964320 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:21:48.964432 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:48.964440 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:48.964776 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:48.965792 (+ 1016us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:48.966076 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.967697 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.970388 (+ 2691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.971882 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.972108 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:48.972332 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:48.972575 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 01:21:48.972806 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18185,"thread_start_us":18016,"threads_started":1}
I20260925 01:21:48.973678 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:48.953059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52646 (local address 127.20.102.129:43149)
0925 01:21:48.965264 (+ 12205us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:48.965272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:48.965300 (+ 28us) server_negotiation.cc:407] Connection header received
0925 01:21:48.965833 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:48.965845 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:48.966067 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:48.967022 (+ 955us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:48.967249 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.968387 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.970823 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.972263 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.972866 (+ 603us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:48.973070 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:48.973142 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 01:21:48.973301 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12026,"thread_start_us":11883,"threads_started":1}
I20260925 01:21:48.973559 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:48.945446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43149 (local address 127.0.0.1:52646)
0925 01:21:48.955615 (+ 10169us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:48.955637 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:48.955777 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:48.966286 (+ 10509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:48.966292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:48.966319 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:48.966728 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:48.966741 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.968538 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.968543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:48.970599 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:48.970611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.972592 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.972604 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:48.972803 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:48.972845 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 01:21:48.972926 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 01:21:48.973147 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10029,"mutex_wait_us":68,"thread_start_us":18830,"threads_started":1}
I20260925 01:21:48.973575 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:48.945208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:44757 (local address 127.0.0.1:46388)
0925 01:21:48.946245 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:48.946292 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:48.946470 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:48.965064 (+ 18594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:48.965071 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:48.965111 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:48.965551 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:48.965566 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.967841 (+ 2275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.967849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:48.970177 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:48.970194 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.972117 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.972127 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:48.972321 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:48.972370 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 01:21:48.972521 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 01:21:48.973167 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":45,"thread_start_us":991,"threads_started":1}
I20260925 01:21:48.974514 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:48.946917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:43983 (local address 127.0.0.1:44658)
0925 01:21:48.964554 (+ 17637us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:48.964574 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:48.964732 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:48.965394 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:48.965399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:48.965428 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:48.965896 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:48.965908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.967482 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.967490 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:48.970898 (+ 3408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:48.970907 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.972526 (+ 1619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.972534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:48.972760 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:48.973974 (+ 1214us) client_negotiation.cc:769] Sending connection context
0925 01:21:48.974060 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 01:21:48.974198 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17509,"mutex_wait_us":47,"thread_start_us":15721,"threads_started":1}
I20260925 01:21:48.975303 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:48.955524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44658 (local address 127.20.102.130:43983)
0925 01:21:48.956196 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:48.956205 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:48.964745 (+ 8540us) server_negotiation.cc:407] Connection header received
0925 01:21:48.964890 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:48.964902 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:48.965169 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:48.965391 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:48.966088 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.967339 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.971062 (+ 3723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:48.972063 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:48.972226 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:48.972430 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:48.974733 (+ 2303us) server_negotiation.cc:299] Negotiation successful
0925 01:21:48.974915 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":2296,"threads_started":1}
I20260925 01:21:48.977916 8661 tablet_service.cc:1467] Processing CreateTablet for tablet 47a722cb26324e11b2e0692b8fa55a08 (DEFAULT_TABLE table=test-table [id=c7cf8c284d04427d81b8bd2cb3e1f3ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:48.978490 8400 tablet_service.cc:1467] Processing CreateTablet for tablet 47a722cb26324e11b2e0692b8fa55a08 (DEFAULT_TABLE table=test-table [id=c7cf8c284d04427d81b8bd2cb3e1f3ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:48.979346 8661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a722cb26324e11b2e0692b8fa55a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:48.979009 8531 tablet_service.cc:1467] Processing CreateTablet for tablet 47a722cb26324e11b2e0692b8fa55a08 (DEFAULT_TABLE table=test-table [id=c7cf8c284d04427d81b8bd2cb3e1f3ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 01:21:48.979941 8400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a722cb26324e11b2e0692b8fa55a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:48.980348 8531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a722cb26324e11b2e0692b8fa55a08. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:48.982410 8465 heartbeater.cc:503] Master 127.20.102.190:36927 was elected leader, sending a full tablet report...
I20260925 01:21:48.991955 8744 tablet_bootstrap.cc:492] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea: Bootstrap starting.
I20260925 01:21:48.995648 8744 tablet_bootstrap.cc:654] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:48.996800 8744 log.cc:824] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:48.997459 8746 tablet_bootstrap.cc:492] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af: Bootstrap starting.
I20260925 01:21:48.999336 8745 tablet_bootstrap.cc:492] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb: Bootstrap starting.
I20260925 01:21:49.001804 8746 tablet_bootstrap.cc:654] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:49.003423 8745 tablet_bootstrap.cc:654] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:49.003556 8744 tablet_bootstrap.cc:492] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea: No bootstrap required, opened a new log
I20260925 01:21:49.003837 8744 ts_tablet_manager.cc:1397] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260925 01:21:49.003890 8746 log.cc:824] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:49.004568 8745 log.cc:824] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:49.007141 8746 tablet_bootstrap.cc:492] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af: No bootstrap required, opened a new log
I20260925 01:21:49.007408 8746 ts_tablet_manager.cc:1397] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260925 01:21:49.009253 8745 tablet_bootstrap.cc:492] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb: No bootstrap required, opened a new log
I20260925 01:21:49.009599 8745 ts_tablet_manager.cc:1397] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 01:21:49.014727 8744 raft_consensus.cc:348] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f0a9cf9c264bbf876102dea0d5affb" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43149 } } peers { permanent_uuid: "58a1a382f58b46bd8fefe05acfcdadea" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44757 } } peers { permanent_uuid: "b3dd999e8a0647ad94d654b086cbf5af" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43983 } }
I20260925 01:21:49.015205 8744 raft_consensus.cc:374] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:49.015327 8744 raft_consensus.cc:729] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a1a382f58b46bd8fefe05acfcdadea, State: Initialized, Role: FOLLOWER
I20260925 01:21:49.015964 8744 consensus_queue.cc:260] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f0a9cf9c264bbf876102dea0d5affb" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43149 } } peers { permanent_uuid: "58a1a382f58b46bd8fefe05acfcdadea" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44757 } } peers { permanent_uuid: "b3dd999e8a0647ad94d654b086cbf5af" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43983 } }
I20260925 01:21:49.017190 8726 heartbeater.cc:503] Master 127.20.102.190:36927 was elected leader, sending a full tablet report...
I20260925 01:21:49.017833 8744 ts_tablet_manager.cc:1428] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260925 01:21:49.018570 8745 raft_consensus.cc:348] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f0a9cf9c264bbf876102dea0d5affb" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43149 } } peers { permanent_uuid: "58a1a382f58b46bd8fefe05acfcdadea" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44757 } } peers { permanent_uuid: "b3dd999e8a0647ad94d654b086cbf5af" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43983 } }
I20260925 01:21:49.018625 8746 raft_consensus.cc:348] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f0a9cf9c264bbf876102dea0d5affb" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43149 } } peers { permanent_uuid: "58a1a382f58b46bd8fefe05acfcdadea" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44757 } } peers { permanent_uuid: "b3dd999e8a0647ad94d654b086cbf5af" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43983 } }
I20260925 01:21:49.019003 8745 raft_consensus.cc:374] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:49.021054 8746 raft_consensus.cc:374] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:49.021063 8745 raft_consensus.cc:729] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95f0a9cf9c264bbf876102dea0d5affb, State: Initialized, Role: FOLLOWER
I20260925 01:21:49.021171 8746 raft_consensus.cc:729] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3dd999e8a0647ad94d654b086cbf5af, State: Initialized, Role: FOLLOWER
I20260925 01:21:49.021725 8745 consensus_queue.cc:260] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f0a9cf9c264bbf876102dea0d5affb" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43149 } } peers { permanent_uuid: "58a1a382f58b46bd8fefe05acfcdadea" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44757 } } peers { permanent_uuid: "b3dd999e8a0647ad94d654b086cbf5af" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43983 } }
I20260925 01:21:49.021806 8746 consensus_queue.cc:260] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f0a9cf9c264bbf876102dea0d5affb" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43149 } } peers { permanent_uuid: "58a1a382f58b46bd8fefe05acfcdadea" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44757 } } peers { permanent_uuid: "b3dd999e8a0647ad94d654b086cbf5af" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43983 } }
I20260925 01:21:49.023455 8596 heartbeater.cc:503] Master 127.20.102.190:36927 was elected leader, sending a full tablet report...
I20260925 01:21:49.023553 8745 ts_tablet_manager.cc:1428] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260925 01:21:49.024961 8746 ts_tablet_manager.cc:1428] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af: Time spent starting tablet: real 0.017s user 0.008s sys 0.006s
W20260925 01:21:49.138932 8727 tablet.cc:2369] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:21:49.177294 8597 tablet.cc:2369] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 01:21:49.206827 8466 tablet.cc:2369] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:21:49.216470 8751 raft_consensus.cc:482] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 01:21:49.216967 8751 raft_consensus.cc:504] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f0a9cf9c264bbf876102dea0d5affb" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43149 } } peers { permanent_uuid: "58a1a382f58b46bd8fefe05acfcdadea" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44757 } } peers { permanent_uuid: "b3dd999e8a0647ad94d654b086cbf5af" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43983 } }
I20260925 01:21:49.218818 8751 leader_election.cc:288] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58a1a382f58b46bd8fefe05acfcdadea (127.20.102.131:44757), b3dd999e8a0647ad94d654b086cbf5af (127.20.102.130:43983)
I20260925 01:21:49.225672 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.219436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:55405 (local address 127.20.102.130:43983)
0925 01:21:49.219739 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:49.219749 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:49.220375 (+ 626us) server_negotiation.cc:407] Connection header received
0925 01:21:49.220612 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:49.220619 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:49.220756 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:49.220855 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:49.221441 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.222141 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.223957 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.224780 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.224949 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:49.225024 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:49.225291 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 01:21:49.225417 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 01:21:49.225718 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.219188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.130:43983 (local address 127.20.102.129:55405)
0925 01:21:49.220250 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:49.220271 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:49.220446 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:49.220888 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:49.220893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:49.220930 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:49.221297 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:49.221307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.222269 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.222276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:49.223776 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:49.223788 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.224954 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.224960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:49.225129 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:49.225153 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 01:21:49.225262 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 01:21:49.225390 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":969,"threads_started":1}
I20260925 01:21:49.225889 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.219296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.20.102.129:57175 (local address 127.20.102.131:44757)
0925 01:21:49.219534 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:49.219545 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:49.220587 (+ 1042us) server_negotiation.cc:407] Connection header received
0925 01:21:49.220822 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:49.220827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:49.220910 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:49.221024 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 01:21:49.221668 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.222367 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.224162 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.225004 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.225204 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:49.225273 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:49.225520 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 01:21:49.225649 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 01:21:49.225986 8753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.219235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.131:44757 (local address 127.20.102.129:57175)
0925 01:21:49.220479 (+ 1244us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:49.220493 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 01:21:49.220651 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:21:49.221041 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:21:49.221045 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:21:49.221085 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 01:21:49.221482 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:49.221491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.222523 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.222528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:49.223977 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:21:49.223989 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.225204 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.225210 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:21:49.225367 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:49.225388 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 01:21:49.225492 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 01:21:49.225622 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"thread_start_us":525,"threads_started":1}
I20260925 01:21:49.226931 8551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a722cb26324e11b2e0692b8fa55a08" candidate_uuid: "95f0a9cf9c264bbf876102dea0d5affb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3dd999e8a0647ad94d654b086cbf5af" is_pre_election: true
I20260925 01:21:49.227026 8681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a722cb26324e11b2e0692b8fa55a08" candidate_uuid: "95f0a9cf9c264bbf876102dea0d5affb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58a1a382f58b46bd8fefe05acfcdadea" is_pre_election: true
I20260925 01:21:49.227586 8681 raft_consensus.cc:2454] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95f0a9cf9c264bbf876102dea0d5affb in term 0.
I20260925 01:21:49.227578 8551 raft_consensus.cc:2454] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95f0a9cf9c264bbf876102dea0d5affb in term 0.
I20260925 01:21:49.228372 8354 leader_election.cc:302] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 58a1a382f58b46bd8fefe05acfcdadea, 95f0a9cf9c264bbf876102dea0d5affb, b3dd999e8a0647ad94d654b086cbf5af; no voters:
I20260925 01:21:49.228830 8751 raft_consensus.cc:2781] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 01:21:49.228982 8751 raft_consensus.cc:482] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 01:21:49.229096 8751 raft_consensus.cc:3037] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:49.231282 8751 raft_consensus.cc:504] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f0a9cf9c264bbf876102dea0d5affb" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43149 } } peers { permanent_uuid: "58a1a382f58b46bd8fefe05acfcdadea" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44757 } } peers { permanent_uuid: "b3dd999e8a0647ad94d654b086cbf5af" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43983 } }
I20260925 01:21:49.232108 8751 leader_election.cc:288] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [CANDIDATE]: Term 1 election: Requested vote from peers 58a1a382f58b46bd8fefe05acfcdadea (127.20.102.131:44757), b3dd999e8a0647ad94d654b086cbf5af (127.20.102.130:43983)
I20260925 01:21:49.232630 8681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a722cb26324e11b2e0692b8fa55a08" candidate_uuid: "95f0a9cf9c264bbf876102dea0d5affb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58a1a382f58b46bd8fefe05acfcdadea"
I20260925 01:21:49.232725 8551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a722cb26324e11b2e0692b8fa55a08" candidate_uuid: "95f0a9cf9c264bbf876102dea0d5affb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3dd999e8a0647ad94d654b086cbf5af"
I20260925 01:21:49.232906 8681 raft_consensus.cc:3037] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:49.232916 8551 raft_consensus.cc:3037] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:49.236253 8551 raft_consensus.cc:2454] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95f0a9cf9c264bbf876102dea0d5affb in term 1.
I20260925 01:21:49.236253 8681 raft_consensus.cc:2454] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95f0a9cf9c264bbf876102dea0d5affb in term 1.
I20260925 01:21:49.236661 8354 leader_election.cc:302] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58a1a382f58b46bd8fefe05acfcdadea, 95f0a9cf9c264bbf876102dea0d5affb; no voters:
I20260925 01:21:49.237058 8751 raft_consensus.cc:2781] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:49.237525 8751 raft_consensus.cc:686] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [term 1 LEADER]: Becoming Leader. State: Replica: 95f0a9cf9c264bbf876102dea0d5affb, State: Running, Role: LEADER
I20260925 01:21:49.238023 8751 consensus_queue.cc:237] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f0a9cf9c264bbf876102dea0d5affb" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43149 } } peers { permanent_uuid: "58a1a382f58b46bd8fefe05acfcdadea" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44757 } } peers { permanent_uuid: "b3dd999e8a0647ad94d654b086cbf5af" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43983 } }
I20260925 01:21:49.243053 8278 catalog_manager.cc:5495] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb reported cstate change: term changed from 0 to 1, leader changed from <none> to 95f0a9cf9c264bbf876102dea0d5affb (127.20.102.129). New cstate: current_term: 1 leader_uuid: "95f0a9cf9c264bbf876102dea0d5affb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95f0a9cf9c264bbf876102dea0d5affb" member_type: VOTER last_known_addr { host: "127.20.102.129" port: 43149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58a1a382f58b46bd8fefe05acfcdadea" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dd999e8a0647ad94d654b086cbf5af" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43983 } health_report { overall_health: UNKNOWN } } }
I20260925 01:21:49.271178 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.265438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52648 (local address 127.20.102.129:43149)
0925 01:21:49.265657 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:49.265662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:49.265680 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:21:49.265857 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:49.265861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:49.265960 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:49.266085 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:21:49.266445 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.267139 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.268810 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.269231 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.269329 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:49.270023 (+ 694us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:21:49.270563 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:21:49.270680 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:49.270785 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 01:21:49.270856 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 01:21:49.279858 8681 raft_consensus.cc:1261] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea [term 1 FOLLOWER]: Refusing update from remote peer 95f0a9cf9c264bbf876102dea0d5affb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:21:49.279857 8551 raft_consensus.cc:1261] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af [term 1 FOLLOWER]: Refusing update from remote peer 95f0a9cf9c264bbf876102dea0d5affb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:21:49.280962 8755 consensus_queue.cc:1035] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dd999e8a0647ad94d654b086cbf5af" member_type: VOTER last_known_addr { host: "127.20.102.130" port: 43983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:21:49.281400 8751 consensus_queue.cc:1035] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a1a382f58b46bd8fefe05acfcdadea" member_type: VOTER last_known_addr { host: "127.20.102.131" port: 44757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:21:49.303007 8759 mvcc.cc:205] Tried to move back new op lower bound from 7333065970799763456 to 7333065970642640896. Current Snapshot: MvccSnapshot[applied={T|T < 7333065970799763456}]
I20260925 01:21:49.314493 8279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:21:49.314810 8279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 01:21:49.318531 8279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 628a84eabefa4d609a99b05b9fabe731: Sending DeleteTablet for 3 replicas of tablet 47a722cb26324e11b2e0692b8fa55a08
I20260925 01:21:49.319800 8400 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a722cb26324e11b2e0692b8fa55a08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:49 UTC) from {username='slave'} at 127.0.0.1:52646
I20260925 01:21:49.319792 8661 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a722cb26324e11b2e0692b8fa55a08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:49 UTC) from {username='slave'} at 127.0.0.1:46388
I20260925 01:21:49.319921 8531 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a722cb26324e11b2e0692b8fa55a08 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 01:21:49 UTC) from {username='slave'} at 127.0.0.1:44658
I20260925 01:21:49.324818 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 8338
I20260925 01:21:49.332085 8768 tablet_replica.cc:330] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb: stopping tablet replica
I20260925 01:21:49.332614 8768 raft_consensus.cc:2229] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [term 1 LEADER]: Raft consensus shutting down.
I20260925 01:21:49.335337 8768 raft_consensus.cc:2258] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:49.337383 8769 tablet_replica.cc:330] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea: stopping tablet replica
I20260925 01:21:49.337834 8769 raft_consensus.cc:2229] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:21:49.338466 8768 ts_tablet_manager.cc:1905] T 47a722cb26324e11b2e0692b8fa55a08 P 95f0a9cf9c264bbf876102dea0d5affb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:49.338927 8770 tablet_replica.cc:330] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af: stopping tablet replica
I20260925 01:21:49.338945 8769 raft_consensus.cc:2258] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:49.341010 8770 raft_consensus.cc:2229] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 01:21:49.341214 8769 ts_tablet_manager.cc:1905] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:49.342028 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.326890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.20.102.129:43149)
0925 01:21:49.327070 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:49.327076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:49.330140 (+ 3064us) server_negotiation.cc:407] Connection header received
0925 01:21:49.331161 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:49.331170 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:49.331307 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:49.331401 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:49.332328 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.333832 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.336399 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.337255 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.337362 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:49.339510 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:49.339525 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:49.339542 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:49.340393 (+ 851us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:49.340462 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:49.340488 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:49.340570 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:49.341482 (+ 912us) server_negotiation.cc:299] Negotiation successful
0925 01:21:49.341603 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:21:49.343573 8460 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:49.344220 8770 raft_consensus.cc:2258] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 01:21:49.346637 8770 ts_tablet_manager.cc:1905] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 01:21:49.348304 8769 ts_tablet_manager.cc:1918] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:49.348466 8769 log.cc:1196] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-2/wal/wals/47a722cb26324e11b2e0692b8fa55a08
I20260925 01:21:49.349035 8769 ts_tablet_manager.cc:1939] T 47a722cb26324e11b2e0692b8fa55a08 P 58a1a382f58b46bd8fefe05acfcdadea: Deleting consensus metadata
I20260925 01:21:49.350950 8264 catalog_manager.cc:4841] TS 58a1a382f58b46bd8fefe05acfcdadea (127.20.102.131:44757): tablet 47a722cb26324e11b2e0692b8fa55a08 (table test-table [id=c7cf8c284d04427d81b8bd2cb3e1f3ef]) successfully deleted
I20260925 01:21:49.353863 8770 ts_tablet_manager.cc:1918] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 01:21:49.354075 8770 log.cc:1196] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af: Deleting WAL directory at /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790298989998230-20890-0/minicluster-data/ts-1/wal/wals/47a722cb26324e11b2e0692b8fa55a08
I20260925 01:21:49.354591 8770 ts_tablet_manager.cc:1939] T 47a722cb26324e11b2e0692b8fa55a08 P b3dd999e8a0647ad94d654b086cbf5af: Deleting consensus metadata
I20260925 01:21:49.356091 8263 catalog_manager.cc:4841] TS b3dd999e8a0647ad94d654b086cbf5af (127.20.102.130:43983): tablet 47a722cb26324e11b2e0692b8fa55a08 (table test-table [id=c7cf8c284d04427d81b8bd2cb3e1f3ef]) successfully deleted
I20260925 01:21:49.456693 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 8338
I20260925 01:21:49.476863 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 8469
W20260925 01:21:49.476856 8265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.20.102.129:43149 (error 108)
I20260925 01:21:49.479519 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.478434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43149 (local address 127.0.0.1:52668)
0925 01:21:49.479233 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:49.479348 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":589,"threads_started":1}
W20260925 01:21:49.479794 8265 catalog_manager.cc:4584] TS 95f0a9cf9c264bbf876102dea0d5affb (127.20.102.129:43149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 47a722cb26324e11b2e0692b8fa55a08: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43149: connect: Connection refused (error 111)
I20260925 01:21:49.484238 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.477776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44672 (local address 127.20.102.130:43983)
0925 01:21:49.477960 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:49.477967 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:49.478010 (+ 43us) server_negotiation.cc:407] Connection header received
0925 01:21:49.478243 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:49.478248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:49.478359 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:49.478465 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:49.478947 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.479884 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.481351 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.481743 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.481835 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:49.482939 (+ 1104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:49.482948 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:49.482962 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:49.483559 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:49.483628 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:49.483647 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:49.483737 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:49.483902 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 01:21:49.483988 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:21:49.484987 8591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:49.533578 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.533005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43149 (local address 127.0.0.1:52680)
0925 01:21:49.533244 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:49.533358 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 01:21:49.534674 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.534257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43149 (local address 127.0.0.1:52694)
0925 01:21:49.534426 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:49.534510 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 01:21:49.578718 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 8469
I20260925 01:21:49.595217 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.594725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43149 (local address 127.0.0.1:52702)
0925 01:21:49.594942 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:49.595043 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 01:21:49.596378 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.595916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43149 (local address 127.0.0.1:52710)
0925 01:21:49.596095 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:49.596165 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 01:21:49.596947 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 8599
I20260925 01:21:49.603942 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.597751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46404 (local address 127.20.102.131:44757)
0925 01:21:49.597942 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:49.597947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:49.597966 (+ 19us) server_negotiation.cc:407] Connection header received
0925 01:21:49.598113 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:49.598118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:49.598204 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:49.598289 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:49.598709 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.599399 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.600902 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:49.601312 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:49.601417 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:49.602600 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:49.602610 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:49.602626 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:49.603193 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:49.603242 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:49.603262 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:49.603349 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:49.603478 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 01:21:49.603598 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 01:21:49.604602 8721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:49.677191 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.676622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43149 (local address 127.0.0.1:52716)
0925 01:21:49.676920 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:49.677042 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260925 01:21:49.678413 8774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:49.677937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.129:43149 (local address 127.0.0.1:52724)
0925 01:21:49.678211 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 01:21:49.678279 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.129:43149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260925 01:21:49.693604 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 8599
I20260925 01:21:49.713660 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 8248
I20260925 01:21:49.714998 8309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:49.827696 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 8248
2026-09-25T01:21:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2983 ms)
[----------] 6 tests from EncryptionPolicyTest (22085 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:21:49 dist-test-slave-s12s krb5kdc[8780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:21:49 dist-test-slave-s12s krb5kdc[8780](info): set up 2 sockets
Sep 25 01:21:49 dist-test-slave-s12s krb5kdc[8780](info): commencing operation
krb5kdc: starting...
W20260925 01:21:50.950726 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:21:51 dist-test-slave-s12s krb5kdc[8780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:21:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:21:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:21:51.212785 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:51.567919 8793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:51.568254 8793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:51.568367 8793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:51.577293 8793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:21:51.577430 8793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:51.577507 8793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:51.577656 8793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:21:51.577749 8793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:21:51.589488 8793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:51.591207 8793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:51.592487 8793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:51.604586 8798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:51.604693 8799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:51.604614 8801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:51.605427 8793 server_base.cc:1034] running on GCE node
I20260925 01:21:51.606290 8793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:51.608192 8793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:51.609421 8793 hybrid_clock.cc:648] HybridClock initialized: now 1790299311609366 us; error 50 us; skew 500 ppm
I20260925 01:21:51.609810 8793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:51 dist-test-slave-s12s krb5kdc[8780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299311, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:51.613409 8793 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:21:51.615427 8793 webserver.cc:460] Webserver started at http://127.0.0.1:36693/ using document root <none> and password file <none>
I20260925 01:21:51.616436 8793 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:51.616611 8793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:51.617094 8793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:51.621569 8793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "b6d70cfbd3404ffbb04f572e231a36ab"
format_stamp: "Formatted at 2026-09-25 01:21:51 on dist-test-slave-s12s"
server_key: "f28b34a333713c0252423eca7fb8009e"
server_key_iv: "609ba006329bfe6591a34fa62f6a2e3a"
server_key_version: "encryptionkey@0"
I20260925 01:21:51.622464 8793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "b6d70cfbd3404ffbb04f572e231a36ab"
format_stamp: "Formatted at 2026-09-25 01:21:51 on dist-test-slave-s12s"
server_key: "f28b34a333713c0252423eca7fb8009e"
server_key_iv: "609ba006329bfe6591a34fa62f6a2e3a"
server_key_version: "encryptionkey@0"
I20260925 01:21:51.627709 8793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:21:51.631206 8808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:51.632532 8793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:21:51.632776 8793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "b6d70cfbd3404ffbb04f572e231a36ab"
format_stamp: "Formatted at 2026-09-25 01:21:51 on dist-test-slave-s12s"
server_key: "f28b34a333713c0252423eca7fb8009e"
server_key_iv: "609ba006329bfe6591a34fa62f6a2e3a"
server_key_version: "encryptionkey@0"
I20260925 01:21:51.632973 8793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:51.650739 8793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:51.655169 8793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:51.655478 8793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:51.674338 8793 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34257
I20260925 01:21:51.674357 8859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34257 every 8 connection(s)
I20260925 01:21:51.675786 8793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:21:51.679522 8860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:51.679694 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 8793
I20260925 01:21:51.679922 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:21:51.680581 20890 external_mini_cluster.cc:1428] Setting key d8a11e89195b1628786814e055922ab4
I20260925 01:21:51.694198 8860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab: Bootstrap starting.
Sep 25 01:21:51 dist-test-slave-s12s krb5kdc[8780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 01:21:51.697360 8860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:51.698796 8860 log.cc:824] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:51.702553 8860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab: No bootstrap required, opened a new log
I20260925 01:21:51.707423 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:51.682551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48377 (local address 127.0.0.1:34257)
0925 01:21:51.683372 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:51.683386 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:51.683444 (+ 58us) server_negotiation.cc:407] Connection header received
0925 01:21:51.685554 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:51.685583 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:51.686111 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:51.686561 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:51.687920 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:51.689827 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:51.691139 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:51.692247 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:51.692442 (+ 195us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:51.697934 (+ 5492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:51.697946 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:51.697965 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:51.698010 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:51.702321 (+ 4311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:51.703282 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:51.703291 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:51.703298 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:51.703508 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:51.704045 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:51.704050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:51.704053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:51.704836 (+ 783us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:51.705168 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:51.705649 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 01:21:51.706015 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":330,"threads_started":1}
I20260925 01:21:51.711869 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:21:51.712554 8860 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d70cfbd3404ffbb04f572e231a36ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34257 } }
I20260925 01:21:51.712960 8860 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:51.713060 8860 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d70cfbd3404ffbb04f572e231a36ab, State: Initialized, Role: FOLLOWER
I20260925 01:21:51.713697 8860 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d70cfbd3404ffbb04f572e231a36ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34257 } }
I20260925 01:21:51.713932 8860 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:21:51.714062 8860 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:21:51.714241 8860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:51.716960 8860 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d70cfbd3404ffbb04f572e231a36ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34257 } }
I20260925 01:21:51.717590 8860 leader_election.cc:302] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6d70cfbd3404ffbb04f572e231a36ab; no voters:
I20260925 01:21:51.718288 8860 leader_election.cc:288] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:21:51.718401 8870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:51.721012 8860 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:21:51.722464 8870 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [term 1 LEADER]: Becoming Leader. State: Replica: b6d70cfbd3404ffbb04f572e231a36ab, State: Running, Role: LEADER
I20260925 01:21:51.723215 8870 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d70cfbd3404ffbb04f572e231a36ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34257 } }
I20260925 01:21:51.726271 8871 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6d70cfbd3404ffbb04f572e231a36ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d70cfbd3404ffbb04f572e231a36ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34257 } } }
I20260925 01:21:51.726501 8872 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6d70cfbd3404ffbb04f572e231a36ab. Latest consensus state: current_term: 1 leader_uuid: "b6d70cfbd3404ffbb04f572e231a36ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6d70cfbd3404ffbb04f572e231a36ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34257 } } }
I20260925 01:21:51.726694 8871 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:51.726768 8872 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:51.727618 8880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:21:51.738400 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:51.716891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37755 (local address 127.0.0.1:34257)
0925 01:21:51.717061 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:51.717065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:51.717586 (+ 521us) server_negotiation.cc:407] Connection header received
0925 01:21:51.717788 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:51.717805 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:51.717890 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:51.717995 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:51.719849 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:51.720806 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:51.722098 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:51.723006 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:51.723179 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:51.727206 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:51.727220 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:51.727228 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:51.727289 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:51.732218 (+ 4929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:51.733893 (+ 1675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:51.733904 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:51.733911 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:51.734144 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:51.735087 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:51.735098 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:51.735105 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:51.735853 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:51.736306 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:51.737407 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0925 01:21:51.737699 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":51}
I20260925 01:21:51.741366 8880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:21:51.752599 8880 catalog_manager.cc:1347] Generated new cluster ID: b60655e8a5d4455296a08dc00298f5d1
I20260925 01:21:51.752796 8880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:21:51.765769 8880 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:21:51.767316 8880 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:21:51.785254 8880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6d70cfbd3404ffbb04f572e231a36ab: Generated new TSK 0
I20260925 01:21:51.786253 8880 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:21:51.796660 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 8793
I20260925 01:21:51.798800 8855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:51.892755 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 8793
W20260925 01:21:51.910801 8778 connection.cc:535] client connection to 127.0.0.1:34257 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34257): Connection reset by peer (error 104)
2026-09-25T01:21:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2061 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:21:51 dist-test-slave-s12s krb5kdc[8892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:21:51 dist-test-slave-s12s krb5kdc[8892](info): set up 2 sockets
Sep 25 01:21:51 dist-test-slave-s12s krb5kdc[8892](info): commencing operation
krb5kdc: starting...
W20260925 01:21:53.005208 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:21:53 dist-test-slave-s12s krb5kdc[8892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:21:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:21:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:21:53.263885 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:53.601224 8905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:53.601598 8905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:53.601763 8905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:53.609041 8905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:21:53.609163 8905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:53.609210 8905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:53.609301 8905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:21:53.609365 8905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:21:53.619350 8905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36323
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:53.620762 8905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:53.622115 8905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:53.634148 8913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:53.634167 8910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:53.634198 8911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:53.635082 8905 server_base.cc:1034] running on GCE node
I20260925 01:21:53.635807 8905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:53.637255 8905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:53.638449 8905 hybrid_clock.cc:648] HybridClock initialized: now 1790299313638382 us; error 57 us; skew 500 ppm
I20260925 01:21:53.638767 8905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:53 dist-test-slave-s12s krb5kdc[8892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:53.645695 8905 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:21:53.647214 8905 webserver.cc:460] Webserver started at http://127.0.0.1:40709/ using document root <none> and password file <none>
I20260925 01:21:53.647928 8905 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:53.648094 8905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:53.648375 8905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:53.651412 8905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "5ade16a7afcf41aba08138269538ab77"
format_stamp: "Formatted at 2026-09-25 01:21:53 on dist-test-slave-s12s"
server_key: "017e0d9637d19a9897324ba07c0ac1ac"
server_key_iv: "6fe5786773c1f6de69c4cd0a558c4c44"
server_key_version: "encryptionkey@0"
I20260925 01:21:53.652130 8905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "5ade16a7afcf41aba08138269538ab77"
format_stamp: "Formatted at 2026-09-25 01:21:53 on dist-test-slave-s12s"
server_key: "017e0d9637d19a9897324ba07c0ac1ac"
server_key_iv: "6fe5786773c1f6de69c4cd0a558c4c44"
server_key_version: "encryptionkey@0"
I20260925 01:21:53.657213 8905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:21:53.660548 8920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:53.661911 8905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:21:53.662096 8905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "5ade16a7afcf41aba08138269538ab77"
format_stamp: "Formatted at 2026-09-25 01:21:53 on dist-test-slave-s12s"
server_key: "017e0d9637d19a9897324ba07c0ac1ac"
server_key_iv: "6fe5786773c1f6de69c4cd0a558c4c44"
server_key_version: "encryptionkey@0"
I20260925 01:21:53.662287 8905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:53.682828 8905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:53.687059 8905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:53.687352 8905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:53.706425 8905 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36323
I20260925 01:21:53.706477 8971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36323 every 8 connection(s)
I20260925 01:21:53.707839 8905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:21:53.709406 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 8905
I20260925 01:21:53.709600 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:21:53.710191 20890 external_mini_cluster.cc:1428] Setting key 2b5427bc1dfbb0b2bd18618a5620eb86
I20260925 01:21:53.712523 8972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:53.725160 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:53.712070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37107 (local address 127.0.0.1:36323)
0925 01:21:53.713164 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:53.713178 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:53.713225 (+ 47us) server_negotiation.cc:407] Connection header received
0925 01:21:53.715002 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:53.715024 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:53.715460 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:53.715832 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:53.716452 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:53.718055 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:53.719965 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:53.720785 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:53.720924 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:53.722368 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:53.722378 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:53.722393 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:53.723104 (+ 711us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:53.723194 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:53.723252 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:53.723419 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:53.723809 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 01:21:53.724008 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":577,"threads_started":1}
I20260925 01:21:53.725560 8972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77: Bootstrap starting.
I20260925 01:21:53.729048 8972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:53.729315 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:21:53.730108 8972 log.cc:824] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:53.733201 8972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77: No bootstrap required, opened a new log
I20260925 01:21:53.741541 8975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:53.734289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52581 (local address 127.0.0.1:36323)
0925 01:21:53.734466 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:53.734473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:53.735033 (+ 560us) server_negotiation.cc:407] Connection header received
0925 01:21:53.735218 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:53.735224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:53.735310 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:53.735417 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:53.735959 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:53.737171 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:53.738635 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:53.739356 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:53.739489 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:53.740660 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:53.740667 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:53.740672 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:53.740719 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:53.740784 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:53.740811 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:53.740933 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:53.741117 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 01:21:53.741169 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:21:53.742844 8972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ade16a7afcf41aba08138269538ab77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260925 01:21:53.743388 8972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:53.743502 8972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ade16a7afcf41aba08138269538ab77, State: Initialized, Role: FOLLOWER
I20260925 01:21:53.744071 8972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ade16a7afcf41aba08138269538ab77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260925 01:21:53.744277 8972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:21:53.744374 8972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:21:53.744504 8972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:53.746718 8972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ade16a7afcf41aba08138269538ab77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260925 01:21:53.747167 8972 leader_election.cc:302] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ade16a7afcf41aba08138269538ab77; no voters:
I20260925 01:21:53.747773 8972 leader_election.cc:288] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:21:53.747907 8982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:53.749343 8982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [term 1 LEADER]: Becoming Leader. State: Replica: 5ade16a7afcf41aba08138269538ab77, State: Running, Role: LEADER
I20260925 01:21:53.749926 8982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ade16a7afcf41aba08138269538ab77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } }
I20260925 01:21:53.750082 8972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:21:53.752965 8983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ade16a7afcf41aba08138269538ab77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ade16a7afcf41aba08138269538ab77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } } }
I20260925 01:21:53.753432 8983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:53.753304 8984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ade16a7afcf41aba08138269538ab77. Latest consensus state: current_term: 1 leader_uuid: "5ade16a7afcf41aba08138269538ab77" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ade16a7afcf41aba08138269538ab77" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36323 } } }
I20260925 01:21:53.753657 8984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:53.757066 8988 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:21:53.763715 8988 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:21:53.775759 8988 catalog_manager.cc:1347] Generated new cluster ID: dd6c06677228467e8885be489e643681
I20260925 01:21:53.775875 8988 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:21:53.787883 8988 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:21:53.789981 8988 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:21:53.797235 8988 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ade16a7afcf41aba08138269538ab77: Generated new TSK 0
I20260925 01:21:53.798295 8988 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:21:53.806248 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 8905
W20260925 01:21:53.807427 8967 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37107
W20260925 01:21:53.808179 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 8905: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:21:53.808362 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 8905
W20260925 01:21:53.826077 8890 connection.cc:535] client connection to 127.0.0.1:36323 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36323): Connection reset by peer (error 104)
2026-09-25T01:21:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1912 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:21:53 dist-test-slave-s12s krb5kdc[9003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:21:53 dist-test-slave-s12s krb5kdc[9003](info): set up 2 sockets
Sep 25 01:21:53 dist-test-slave-s12s krb5kdc[9003](info): commencing operation
krb5kdc: starting...
W20260925 01:21:54.909229 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:21:55 dist-test-slave-s12s krb5kdc[9003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:21:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:21:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:21:55.171844 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:55.505926 9016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:55.506246 9016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:55.506343 9016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:55.513676 9016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:21:55.513797 9016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:55.513845 9016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:55.513908 9016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:21:55.513955 9016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:21:55.524116 9016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:55.525432 9016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:55.526721 9016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:55.538630 9021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:55.538586 9022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:55.539047 9024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:55.539516 9016 server_base.cc:1034] running on GCE node
I20260925 01:21:55.540452 9016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:55.542089 9016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:55.543314 9016 hybrid_clock.cc:648] HybridClock initialized: now 1790299315543269 us; error 33 us; skew 500 ppm
I20260925 01:21:55.543697 9016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:55 dist-test-slave-s12s krb5kdc[9003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:55.547348 9016 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:21:55.548949 9016 webserver.cc:460] Webserver started at http://127.0.0.1:35813/ using document root <none> and password file <none>
I20260925 01:21:55.549676 9016 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:55.549786 9016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:55.550060 9016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:55.553524 9016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "b4047ffc6e5447bfa3d3b77287b024c0"
format_stamp: "Formatted at 2026-09-25 01:21:55 on dist-test-slave-s12s"
server_key: "1bf9367efcf351ad2ef8d23af348550f"
server_key_iv: "83f19bd165e118f124b824c32a0b8ef9"
server_key_version: "encryptionkey@0"
I20260925 01:21:55.554350 9016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "b4047ffc6e5447bfa3d3b77287b024c0"
format_stamp: "Formatted at 2026-09-25 01:21:55 on dist-test-slave-s12s"
server_key: "1bf9367efcf351ad2ef8d23af348550f"
server_key_iv: "83f19bd165e118f124b824c32a0b8ef9"
server_key_version: "encryptionkey@0"
I20260925 01:21:55.560151 9016 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:21:55.563973 9031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:55.565469 9016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:21:55.565668 9016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "b4047ffc6e5447bfa3d3b77287b024c0"
format_stamp: "Formatted at 2026-09-25 01:21:55 on dist-test-slave-s12s"
server_key: "1bf9367efcf351ad2ef8d23af348550f"
server_key_iv: "83f19bd165e118f124b824c32a0b8ef9"
server_key_version: "encryptionkey@0"
I20260925 01:21:55.565905 9016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:55.571668 9016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:55.576164 9016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:55.576409 9016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:55.594592 9016 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33115
I20260925 01:21:55.594610 9082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33115 every 8 connection(s)
I20260925 01:21:55.596118 9016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:21:55.596764 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 9016
I20260925 01:21:55.597000 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:21:55.597553 20890 external_mini_cluster.cc:1428] Setting key 31d31c54d6d97b8704d2f810d9627f25
I20260925 01:21:55.600837 9083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:55.607664 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:55.599615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36799 (local address 127.0.0.1:33115)
0925 01:21:55.600620 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:55.600638 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:55.601029 (+ 391us) server_negotiation.cc:407] Connection header received
0925 01:21:55.603556 (+ 2527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:55.603584 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:55.604048 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:55.604509 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:55.605591 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:55.605600 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:55.605618 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:55.605642 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:55.605738 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:55.605763 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:55.605984 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:55.606120 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 01:21:55.606327 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":417,"threads_started":1}
I20260925 01:21:55.611770 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:21:55.615043 9083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0: Bootstrap starting.
I20260925 01:21:55.619462 9083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:55.620023 9086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:55.617264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37819 (local address 127.0.0.1:33115)
0925 01:21:55.617492 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:55.617499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:55.618028 (+ 529us) server_negotiation.cc:407] Connection header received
0925 01:21:55.618206 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:55.618210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:55.618263 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:55.618343 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:55.619304 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:55.619308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:55.619311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:55.619330 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:55.619371 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:55.619385 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:55.619476 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:55.619660 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 01:21:55.619711 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 01:21:55.620862 9083 log.cc:824] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:55.623662 9083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0: No bootstrap required, opened a new log
I20260925 01:21:55.631295 9083 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4047ffc6e5447bfa3d3b77287b024c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260925 01:21:55.631664 9083 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:55.631742 9083 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4047ffc6e5447bfa3d3b77287b024c0, State: Initialized, Role: FOLLOWER
I20260925 01:21:55.632254 9083 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4047ffc6e5447bfa3d3b77287b024c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260925 01:21:55.632471 9083 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:21:55.632652 9083 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:21:55.632900 9083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:55.635118 9083 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4047ffc6e5447bfa3d3b77287b024c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260925 01:21:55.635572 9083 leader_election.cc:302] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4047ffc6e5447bfa3d3b77287b024c0; no voters:
I20260925 01:21:55.636082 9083 leader_election.cc:288] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:21:55.636241 9093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:55.637526 9093 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [term 1 LEADER]: Becoming Leader. State: Replica: b4047ffc6e5447bfa3d3b77287b024c0, State: Running, Role: LEADER
I20260925 01:21:55.638123 9093 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4047ffc6e5447bfa3d3b77287b024c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260925 01:21:55.638342 9083 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:21:55.641356 9094 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4047ffc6e5447bfa3d3b77287b024c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4047ffc6e5447bfa3d3b77287b024c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } } }
I20260925 01:21:55.641328 9095 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4047ffc6e5447bfa3d3b77287b024c0. Latest consensus state: current_term: 1 leader_uuid: "b4047ffc6e5447bfa3d3b77287b024c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4047ffc6e5447bfa3d3b77287b024c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } } }
I20260925 01:21:55.641798 9094 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:55.641845 9095 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:55.643311 9101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:21:55.649457 9101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:21:55.659032 9101 catalog_manager.cc:1347] Generated new cluster ID: 90568c16621c480bbf060e8697550868
I20260925 01:21:55.659164 9101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:21:55.684404 9101 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:21:55.684917 9101 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:21:55.696053 9101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4047ffc6e5447bfa3d3b77287b024c0: Generated new TSK 0
I20260925 01:21:55.697054 9101 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:21:55.701448 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9016
W20260925 01:21:55.702515 9078 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36799
W20260925 01:21:55.703423 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9016: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:21:55.703621 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 9016
2026-09-25T01:21:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1895 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:21:55 dist-test-slave-s12s krb5kdc[9114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:21:55 dist-test-slave-s12s krb5kdc[9114](info): set up 2 sockets
Sep 25 01:21:55 dist-test-slave-s12s krb5kdc[9114](info): commencing operation
krb5kdc: starting...
W20260925 01:21:56.833191 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.009s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:21:57 dist-test-slave-s12s krb5kdc[9114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:21:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:21:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:21:57.096095 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:57.431064 9127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:57.431362 9127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:57.431523 9127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:57.438854 9127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:21:57.438979 9127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:57.439103 9127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:57.439195 9127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:21:57.439276 9127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:21:57.449604 9127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:57.450984 9127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:57.452323 9127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:57.464357 9132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:57.464335 9135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:57.464397 9133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:57.465395 9127 server_base.cc:1034] running on GCE node
I20260925 01:21:57.466214 9127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:57.467671 9127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:57.468894 9127 hybrid_clock.cc:648] HybridClock initialized: now 1790299317468854 us; error 51 us; skew 500 ppm
I20260925 01:21:57.469236 9127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:57 dist-test-slave-s12s krb5kdc[9114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299317, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:57.472311 9127 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:21:57.473924 9127 webserver.cc:460] Webserver started at http://127.0.0.1:44937/ using document root <none> and password file <none>
I20260925 01:21:57.474593 9127 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:57.474746 9127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:57.475044 9127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:57.478226 9127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "51d913032dfd443bba87f75b024123c3"
format_stamp: "Formatted at 2026-09-25 01:21:57 on dist-test-slave-s12s"
server_key: "51a98674ac4ba02f9d7f4471dade53f2"
server_key_iv: "06920b81c07a34d0414d3fbae1542cc5"
server_key_version: "encryptionkey@0"
I20260925 01:21:57.478992 9127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "51d913032dfd443bba87f75b024123c3"
format_stamp: "Formatted at 2026-09-25 01:21:57 on dist-test-slave-s12s"
server_key: "51a98674ac4ba02f9d7f4471dade53f2"
server_key_iv: "06920b81c07a34d0414d3fbae1542cc5"
server_key_version: "encryptionkey@0"
I20260925 01:21:57.484136 9127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 01:21:57.487598 9142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:57.488956 9127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 01:21:57.489146 9127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "51d913032dfd443bba87f75b024123c3"
format_stamp: "Formatted at 2026-09-25 01:21:57 on dist-test-slave-s12s"
server_key: "51a98674ac4ba02f9d7f4471dade53f2"
server_key_iv: "06920b81c07a34d0414d3fbae1542cc5"
server_key_version: "encryptionkey@0"
I20260925 01:21:57.489352 9127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:57.520593 9127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:57.524789 9127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:57.525101 9127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:57.544123 9127 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43633
I20260925 01:21:57.544162 9193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43633 every 8 connection(s)
I20260925 01:21:57.545534 9127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:21:57.549429 9194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:57.551934 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 9127
I20260925 01:21:57.552120 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:21:57.552546 20890 external_mini_cluster.cc:1428] Setting key 7b83ac5e86618a05b7556e5bf0f479d8
I20260925 01:21:57.561781 9194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3: Bootstrap starting.
I20260925 01:21:57.565727 9194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:57.567102 9194 log.cc:824] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:21:57 dist-test-slave-s12s krb5kdc[9114](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 01:21:57.570436 9194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3: No bootstrap required, opened a new log
I20260925 01:21:57.577370 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:57.554532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52161 (local address 127.0.0.1:43633)
0925 01:21:57.555383 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:57.555395 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:57.555437 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:21:57.557176 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:57.557200 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:57.557601 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:57.557931 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:57.559705 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:57.561019 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:57.562793 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:57.563493 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:57.563622 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:57.568822 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:57.568831 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:57.568846 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:57.568885 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:57.572747 (+ 3862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:57.573484 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:57.573509 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:57.573515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:57.573714 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:57.574157 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:57.574163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:57.574172 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:57.574832 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:57.575193 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:57.575725 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 01:21:57.576173 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":392,"threads_started":1}
I20260925 01:21:57.578745 9194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d913032dfd443bba87f75b024123c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43633 } }
I20260925 01:21:57.579150 9194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:57.579272 9194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51d913032dfd443bba87f75b024123c3, State: Initialized, Role: FOLLOWER
I20260925 01:21:57.579927 9194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d913032dfd443bba87f75b024123c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43633 } }
I20260925 01:21:57.580281 9194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:21:57.580438 9194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:21:57.580617 9194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:57.581660 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:21:57.583038 9194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d913032dfd443bba87f75b024123c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43633 } }
I20260925 01:21:57.583520 9194 leader_election.cc:302] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51d913032dfd443bba87f75b024123c3; no voters:
I20260925 01:21:57.584102 9194 leader_election.cc:288] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:21:57.584317 9199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:57.585613 9199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [term 1 LEADER]: Becoming Leader. State: Replica: 51d913032dfd443bba87f75b024123c3, State: Running, Role: LEADER
I20260925 01:21:57.586444 9194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:21:57.586386 9199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d913032dfd443bba87f75b024123c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43633 } }
I20260925 01:21:57.590725 9203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51d913032dfd443bba87f75b024123c3. Latest consensus state: current_term: 1 leader_uuid: "51d913032dfd443bba87f75b024123c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d913032dfd443bba87f75b024123c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43633 } } }
I20260925 01:21:57.591249 9203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:57.592006 9202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51d913032dfd443bba87f75b024123c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51d913032dfd443bba87f75b024123c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43633 } } }
I20260925 01:21:57.592407 9202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:57.602753 9212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:21:57.613493 9212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:21:57.623744 9197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:57.588696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47605 (local address 127.0.0.1:43633)
0925 01:21:57.588951 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:57.588961 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:57.601068 (+ 12107us) server_negotiation.cc:407] Connection header received
0925 01:21:57.601743 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:57.601753 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:57.601865 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:57.601968 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:57.604297 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:57.605517 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:57.608442 (+ 2925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:57.609227 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:57.609345 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:57.613166 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:57.613176 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:57.613185 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:21:57.613254 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:57.618275 (+ 5021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:57.619504 (+ 1229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:57.619513 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:57.619520 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:57.619731 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:21:57.620297 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:21:57.620305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:21:57.620310 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:21:57.620972 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:57.621295 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:57.622852 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0925 01:21:57.623166 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 01:21:57.629040 9212 catalog_manager.cc:1347] Generated new cluster ID: 6cfcaf3758b9432da35fd15850330b7b
I20260925 01:21:57.629174 9212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:21:57.661139 9212 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:21:57.662520 9212 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:21:57.668752 9212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51d913032dfd443bba87f75b024123c3: Generated new TSK 0
I20260925 01:21:57.669610 9212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:21:57.684103 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9127
I20260925 01:21:57.685359 9189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:21:57.779809 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 9127
2026-09-25T01:21:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2077 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:21:57 dist-test-slave-s12s krb5kdc[9227](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:21:57 dist-test-slave-s12s krb5kdc[9227](info): set up 2 sockets
Sep 25 01:21:57 dist-test-slave-s12s krb5kdc[9227](info): commencing operation
krb5kdc: starting...
W20260925 01:21:58.893317 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:21:59 dist-test-slave-s12s krb5kdc[9227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:21:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:21:59.154086 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:21:59.504153 9240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:21:59.504446 9240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:21:59.504626 9240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:21:59.511925 9240 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:21:59.512034 9240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:21:59.512162 9240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:21:59.512274 9240 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:21:59.512349 9240 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:21:59.522434 9240 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:21:59.523793 9240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:21:59.525134 9240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:21:59.537165 9245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:59.537191 9246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:21:59.537317 9248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:21:59.537851 9240 server_base.cc:1034] running on GCE node
I20260925 01:21:59.538730 9240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:21:59.540211 9240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:21:59.541409 9240 hybrid_clock.cc:648] HybridClock initialized: now 1790299319541352 us; error 50 us; skew 500 ppm
I20260925 01:21:59.541762 9240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:21:59 dist-test-slave-s12s krb5kdc[9227](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:21:59.548184 9240 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:21:59.549681 9240 webserver.cc:460] Webserver started at http://127.0.0.1:34185/ using document root <none> and password file <none>
I20260925 01:21:59.550379 9240 fs_manager.cc:359] Metadata directory not provided
I20260925 01:21:59.550508 9240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:21:59.550822 9240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:21:59.553897 9240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "0a6351e8387b422288562727ad5355d0"
format_stamp: "Formatted at 2026-09-25 01:21:59 on dist-test-slave-s12s"
server_key: "41d0947de967e1f97a95e663c5679e2d"
server_key_iv: "ad8a80f737298c3d5f79e4c1f04563ac"
server_key_version: "encryptionkey@0"
I20260925 01:21:59.554662 9240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "0a6351e8387b422288562727ad5355d0"
format_stamp: "Formatted at 2026-09-25 01:21:59 on dist-test-slave-s12s"
server_key: "41d0947de967e1f97a95e663c5679e2d"
server_key_iv: "ad8a80f737298c3d5f79e4c1f04563ac"
server_key_version: "encryptionkey@0"
I20260925 01:21:59.559873 9240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:21:59.563376 9255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:21:59.564790 9240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:21:59.564967 9240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "0a6351e8387b422288562727ad5355d0"
format_stamp: "Formatted at 2026-09-25 01:21:59 on dist-test-slave-s12s"
server_key: "41d0947de967e1f97a95e663c5679e2d"
server_key_iv: "ad8a80f737298c3d5f79e4c1f04563ac"
server_key_version: "encryptionkey@0"
I20260925 01:21:59.565186 9240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:21:59.589030 9240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:21:59.593127 9240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:21:59.593432 9240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:21:59.611884 9240 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37097
I20260925 01:21:59.611918 9306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37097 every 8 connection(s)
I20260925 01:21:59.613755 9240 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:21:59.617683 9307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:21:59.619340 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 9240
I20260925 01:21:59.619529 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:21:59.619987 20890 external_mini_cluster.cc:1428] Setting key 6bfabe57c34dcbd350bfcc49ef4db407
I20260925 01:21:59.632395 9307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0: Bootstrap starting.
I20260925 01:21:59.634616 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:59.621881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41317 (local address 127.0.0.1:37097)
0925 01:21:59.622708 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:59.622724 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:59.622780 (+ 56us) server_negotiation.cc:407] Connection header received
0925 01:21:59.624494 (+ 1714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:59.624523 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:59.625113 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:59.625571 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:59.626070 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:59.628005 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:59.629403 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:59.630480 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:59.630665 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:59.631758 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:59.631770 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:59.631788 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:59.632506 (+ 718us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:59.632620 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:59.632650 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:59.632792 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:59.632987 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 01:21:59.633253 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":330,"threads_started":1}
I20260925 01:21:59.636253 9307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0: Neither blocks nor log segments found. Creating new log.
I20260925 01:21:59.637578 9307 log.cc:824] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0: Log is configured to *not* fsync() on all Append() calls
I20260925 01:21:59.638754 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:21:59.640450 9307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0: No bootstrap required, opened a new log
I20260925 01:21:59.649266 9310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:21:59.643084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58629 (local address 127.0.0.1:37097)
0925 01:21:59.643268 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 01:21:59.643277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:21:59.643635 (+ 358us) server_negotiation.cc:407] Connection header received
0925 01:21:59.643828 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:21:59.643836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:21:59.643894 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:21:59.644007 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:21:59.644500 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:59.645312 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:59.646673 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:21:59.647153 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:21:59.647356 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:21:59.648430 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:21:59.648435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:21:59.648438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:21:59.648455 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:21:59.648490 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:21:59.648504 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:21:59.648597 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 01:21:59.648848 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 01:21:59.648936 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 01:21:59.649006 9307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6351e8387b422288562727ad5355d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260925 01:21:59.649497 9307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:21:59.649617 9307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a6351e8387b422288562727ad5355d0, State: Initialized, Role: FOLLOWER
I20260925 01:21:59.650272 9307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6351e8387b422288562727ad5355d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260925 01:21:59.650521 9307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:21:59.650686 9307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:21:59.650882 9307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:21:59.653206 9307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6351e8387b422288562727ad5355d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260925 01:21:59.653687 9307 leader_election.cc:302] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a6351e8387b422288562727ad5355d0; no voters:
I20260925 01:21:59.654232 9307 leader_election.cc:288] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:21:59.654369 9317 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:21:59.655057 9317 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [term 1 LEADER]: Becoming Leader. State: Replica: 0a6351e8387b422288562727ad5355d0, State: Running, Role: LEADER
I20260925 01:21:59.655666 9317 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6351e8387b422288562727ad5355d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260925 01:21:59.656595 9307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:21:59.658830 9318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a6351e8387b422288562727ad5355d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6351e8387b422288562727ad5355d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } } }
I20260925 01:21:59.658820 9319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a6351e8387b422288562727ad5355d0. Latest consensus state: current_term: 1 leader_uuid: "0a6351e8387b422288562727ad5355d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a6351e8387b422288562727ad5355d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } } }
I20260925 01:21:59.659309 9319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:59.659309 9318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0 [sys.catalog]: This master's current role is: LEADER
I20260925 01:21:59.661482 9324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:21:59.666724 9324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:21:59.675884 9324 catalog_manager.cc:1347] Generated new cluster ID: 9a871c60d9844aad82e263be7009970d
I20260925 01:21:59.676051 9324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:21:59.719980 9324 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:21:59.722015 9324 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:21:59.731948 9324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a6351e8387b422288562727ad5355d0: Generated new TSK 0
I20260925 01:21:59.732875 9324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:21:59.740391 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9240
W20260925 01:21:59.741541 9302 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41317
W20260925 01:21:59.742285 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9240: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:21:59.742447 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 9240
2026-09-25T01:21:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1961 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:21:59 dist-test-slave-s12s krb5kdc[9338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:21:59 dist-test-slave-s12s krb5kdc[9338](info): set up 2 sockets
Sep 25 01:21:59 dist-test-slave-s12s krb5kdc[9338](info): commencing operation
krb5kdc: starting...
W20260925 01:22:00.853225 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:01 dist-test-slave-s12s krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:22:01.120669 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43267
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:01.459252 9351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:01.459564 9351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:01.459657 9351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:01.466856 9351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:01.466977 9351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:01.467023 9351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:01.467080 9351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:01.467123 9351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:01.477548 9351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:01.478972 9351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:01.480406 9351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:01.492928 9357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:01.493017 9356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:01.492969 9359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:01.494225 9351 server_base.cc:1034] running on GCE node
I20260925 01:22:01.495005 9351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:01.496439 9351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:01.497658 9351 hybrid_clock.cc:648] HybridClock initialized: now 1790299321497585 us; error 49 us; skew 500 ppm
I20260925 01:22:01.497985 9351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:01 dist-test-slave-s12s krb5kdc[9338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:01.501140 9351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:22:01.502658 9351 webserver.cc:460] Webserver started at http://127.0.0.1:39805/ using document root <none> and password file <none>
I20260925 01:22:01.503309 9351 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:01.503418 9351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:01.503683 9351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:01.506731 9351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "5e143c1d3d9f4950969d66bafe10463f"
format_stamp: "Formatted at 2026-09-25 01:22:01 on dist-test-slave-s12s"
server_key: "0e8981dc22f717712580eaab9367ec79"
server_key_iv: "0ce0ab4fbdc9b59b06c542cc69222adf"
server_key_version: "encryptionkey@0"
I20260925 01:22:01.507449 9351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "5e143c1d3d9f4950969d66bafe10463f"
format_stamp: "Formatted at 2026-09-25 01:22:01 on dist-test-slave-s12s"
server_key: "0e8981dc22f717712580eaab9367ec79"
server_key_iv: "0ce0ab4fbdc9b59b06c542cc69222adf"
server_key_version: "encryptionkey@0"
I20260925 01:22:01.512778 9351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:22:01.516330 9366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:01.517761 9351 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:22:01.517904 9351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "5e143c1d3d9f4950969d66bafe10463f"
format_stamp: "Formatted at 2026-09-25 01:22:01 on dist-test-slave-s12s"
server_key: "0e8981dc22f717712580eaab9367ec79"
server_key_iv: "0ce0ab4fbdc9b59b06c542cc69222adf"
server_key_version: "encryptionkey@0"
I20260925 01:22:01.518071 9351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:01.523996 9351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:01.528496 9351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:01.528847 9351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:01.548382 9351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43267
I20260925 01:22:01.548432 9417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43267 every 8 connection(s)
I20260925 01:22:01.549861 9351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:01.553741 9418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:01.556972 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 9351
I20260925 01:22:01.557175 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:01.557720 20890 external_mini_cluster.cc:1428] Setting key 24a3abf608dd3d5b0faac081b94dc653
I20260925 01:22:01.565681 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:01.559578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40793 (local address 127.0.0.1:43267)
0925 01:22:01.560365 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:01.560377 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:01.560417 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:22:01.562136 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:01.562160 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:01.562492 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:01.562904 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:01.563985 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:01.563992 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:01.564005 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:01.564024 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:01.564084 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:01.564103 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:01.564357 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:01.564527 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 01:22:01.564763 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":340,"threads_started":1}
I20260925 01:22:01.566308 9418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f: Bootstrap starting.
I20260925 01:22:01.569326 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:01.569751 9418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:01.570801 9418 log.cc:824] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:01.574219 9418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f: No bootstrap required, opened a new log
I20260925 01:22:01.576550 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:01.573919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40535 (local address 127.0.0.1:43267)
0925 01:22:01.574067 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:01.574071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:01.574570 (+ 499us) server_negotiation.cc:407] Connection header received
0925 01:22:01.574781 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:01.574785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:01.574832 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:01.574931 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:01.575909 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:01.575914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:01.575917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:01.575933 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:01.575971 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:01.575988 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:01.576130 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:01.576212 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 01:22:01.576279 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 01:22:01.582559 9418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e143c1d3d9f4950969d66bafe10463f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43267 } }
I20260925 01:22:01.582940 9418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:22:01.583067 9418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e143c1d3d9f4950969d66bafe10463f, State: Initialized, Role: FOLLOWER
I20260925 01:22:01.583655 9418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e143c1d3d9f4950969d66bafe10463f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43267 } }
I20260925 01:22:01.583890 9418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:22:01.584050 9418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:22:01.584215 9418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:01.586479 9418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e143c1d3d9f4950969d66bafe10463f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43267 } }
I20260925 01:22:01.587073 9418 leader_election.cc:302] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e143c1d3d9f4950969d66bafe10463f; no voters:
I20260925 01:22:01.587713 9418 leader_election.cc:288] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:22:01.587806 9428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:01.588528 9428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [term 1 LEADER]: Becoming Leader. State: Replica: 5e143c1d3d9f4950969d66bafe10463f, State: Running, Role: LEADER
I20260925 01:22:01.589210 9428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e143c1d3d9f4950969d66bafe10463f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43267 } }
I20260925 01:22:01.590114 9418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:01.592496 9430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e143c1d3d9f4950969d66bafe10463f. Latest consensus state: current_term: 1 leader_uuid: "5e143c1d3d9f4950969d66bafe10463f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e143c1d3d9f4950969d66bafe10463f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43267 } } }
I20260925 01:22:01.593010 9430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:01.592871 9429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e143c1d3d9f4950969d66bafe10463f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e143c1d3d9f4950969d66bafe10463f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43267 } } }
I20260925 01:22:01.593170 9429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:01.597813 9435 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:22:01.602569 9435 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:01.611209 9435 catalog_manager.cc:1347] Generated new cluster ID: b16d5f48be1743f89bcbee1ff1809f8b
I20260925 01:22:01.611327 9435 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:01.638613 9435 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:01.639182 9435 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:01.648411 9435 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e143c1d3d9f4950969d66bafe10463f: Generated new TSK 0
I20260925 01:22:01.649296 9435 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:22:01.659291 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9351
W20260925 01:22:01.660638 9413 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40793
W20260925 01:22:01.661481 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9351: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:22:01.661640 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 9351
2026-09-25T01:22:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1919 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:01 dist-test-slave-s12s krb5kdc[9449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:01 dist-test-slave-s12s krb5kdc[9449](info): set up 2 sockets
Sep 25 01:22:01 dist-test-slave-s12s krb5kdc[9449](info): commencing operation
krb5kdc: starting...
W20260925 01:22:02.789290 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:02 dist-test-slave-s12s krb5kdc[9449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:22:03.055845 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38059
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:03.391403 9462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:03.391749 9462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:03.391924 9462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:03.399348 9462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:03.399514 9462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:03.399621 9462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:03.399742 9462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:03.399832 9462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:03.410197 9462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:03.411592 9462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:03.412973 9462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:03.425285 9470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:03.425244 9467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:03.425622 9468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:03.426168 9462 server_base.cc:1034] running on GCE node
I20260925 01:22:03.426982 9462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:03.428423 9462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:03.429665 9462 hybrid_clock.cc:648] HybridClock initialized: now 1790299323429591 us; error 88 us; skew 500 ppm
I20260925 01:22:03.430027 9462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:03 dist-test-slave-s12s krb5kdc[9449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:03.433164 9462 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:22:03.434748 9462 webserver.cc:460] Webserver started at http://127.0.0.1:39351/ using document root <none> and password file <none>
I20260925 01:22:03.435447 9462 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:03.435597 9462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:03.435913 9462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:03.439008 9462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "357ab09f3705474494316b159eebf92c"
format_stamp: "Formatted at 2026-09-25 01:22:03 on dist-test-slave-s12s"
server_key: "2101c3ea6bbb1d79f56f2aaa245ade43"
server_key_iv: "c8070472ad66be6dcad339b2400aaf1d"
server_key_version: "encryptionkey@0"
I20260925 01:22:03.439831 9462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "357ab09f3705474494316b159eebf92c"
format_stamp: "Formatted at 2026-09-25 01:22:03 on dist-test-slave-s12s"
server_key: "2101c3ea6bbb1d79f56f2aaa245ade43"
server_key_iv: "c8070472ad66be6dcad339b2400aaf1d"
server_key_version: "encryptionkey@0"
I20260925 01:22:03.445163 9462 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 01:22:03.448764 9477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:03.450169 9462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:22:03.450367 9462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "357ab09f3705474494316b159eebf92c"
format_stamp: "Formatted at 2026-09-25 01:22:03 on dist-test-slave-s12s"
server_key: "2101c3ea6bbb1d79f56f2aaa245ade43"
server_key_iv: "c8070472ad66be6dcad339b2400aaf1d"
server_key_version: "encryptionkey@0"
I20260925 01:22:03.450569 9462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:03.467387 9462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:03.471969 9462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:03.472316 9462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:03.492344 9462 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38059
I20260925 01:22:03.492363 9528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38059 every 8 connection(s)
I20260925 01:22:03.493884 9462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:03.497675 9529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:03.500882 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 9462
I20260925 01:22:03.501073 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:03.501654 20890 external_mini_cluster.cc:1428] Setting key 0b2be9c041913753df4500800e70f469
I20260925 01:22:03.510171 9529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c: Bootstrap starting.
I20260925 01:22:03.513988 9529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:03.515339 9529 log.cc:824] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c: Log is configured to *not* fsync() on all Append() calls
Sep 25 01:22:03 dist-test-slave-s12s krb5kdc[9449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 01:22:03.518644 9529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c: No bootstrap required, opened a new log
I20260925 01:22:03.526748 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:03.503487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:47551 (local address 127.0.0.1:38059)
0925 01:22:03.504243 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:03.504255 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:03.504295 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:22:03.506037 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:03.506060 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:03.506465 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:03.506789 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:03.508573 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:03.509959 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:03.511798 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:03.512472 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:03.512640 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:03.517807 (+ 5167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:03.517819 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:03.517844 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:03.517888 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:03.522062 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:03.522884 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:03.522893 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:03.522902 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:03.523125 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:03.523571 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:03.523576 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:03.523579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:03.524260 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:03.524612 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:03.524983 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 01:22:03.525343 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":346,"threads_started":1}
I20260925 01:22:03.529770 9529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357ab09f3705474494316b159eebf92c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38059 } }
I20260925 01:22:03.530205 9529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:22:03.530346 9529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 357ab09f3705474494316b159eebf92c, State: Initialized, Role: FOLLOWER
I20260925 01:22:03.530984 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:03.530972 9529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357ab09f3705474494316b159eebf92c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38059 } }
I20260925 01:22:03.531272 9529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:22:03.531419 9529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:22:03.531598 9529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:03.533874 9529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357ab09f3705474494316b159eebf92c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38059 } }
I20260925 01:22:03.534379 9529 leader_election.cc:302] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 357ab09f3705474494316b159eebf92c; no voters:
I20260925 01:22:03.534967 9529 leader_election.cc:288] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:22:03.536228 9538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:03.536954 9538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [term 1 LEADER]: Becoming Leader. State: Replica: 357ab09f3705474494316b159eebf92c, State: Running, Role: LEADER
I20260925 01:22:03.537180 9529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:03.537632 9538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357ab09f3705474494316b159eebf92c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38059 } }
I20260925 01:22:03.540994 9539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "357ab09f3705474494316b159eebf92c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357ab09f3705474494316b159eebf92c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38059 } } }
I20260925 01:22:03.541404 9539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:03.541863 9541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 357ab09f3705474494316b159eebf92c. Latest consensus state: current_term: 1 leader_uuid: "357ab09f3705474494316b159eebf92c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "357ab09f3705474494316b159eebf92c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38059 } } }
I20260925 01:22:03.542140 9541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:03.547521 9548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:22:03.554154 9548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:03.555691 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:03.535593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:40063 (local address 127.0.0.1:38059)
0925 01:22:03.535754 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:03.535760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:03.536578 (+ 818us) server_negotiation.cc:407] Connection header received
0925 01:22:03.536658 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:03.536667 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:03.536802 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:03.536915 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:03.538996 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:03.540421 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:03.542408 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:03.543345 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:03.543501 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:03.546577 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:03.546587 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:03.546592 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:03.546640 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:03.550858 (+ 4218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:03.551925 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:03.551930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:03.551933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:03.552084 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:03.552744 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:03.552749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:03.552751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:03.553195 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:03.554895 (+ 1700us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:03.555122 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 01:22:03.555336 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 01:22:03.563702 9548 catalog_manager.cc:1347] Generated new cluster ID: f2d442f1c6dc4e11b6ac3efe288d5749
I20260925 01:22:03.563812 9548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:03.577539 9548 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:03.579200 9548 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:03.590176 9548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 357ab09f3705474494316b159eebf92c: Generated new TSK 0
I20260925 01:22:03.591037 9548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:22:03.603201 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9462
I20260925 01:22:03.604655 9524 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:22:03.695135 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 9462
W20260925 01:22:03.713660 9447 connection.cc:535] client connection to 127.0.0.1:38059 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38059): Connection reset by peer (error 104)
2026-09-25T01:22:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2033 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:03 dist-test-slave-s12s krb5kdc[9561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:03 dist-test-slave-s12s krb5kdc[9561](info): set up 2 sockets
Sep 25 01:22:03 dist-test-slave-s12s krb5kdc[9561](info): commencing operation
krb5kdc: starting...
W20260925 01:22:04.805322 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.008s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:04 dist-test-slave-s12s krb5kdc[9561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:22:05.070346 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44963
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:05.404636 9574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:05.404971 9574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:05.405162 9574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:05.412567 9574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:05.412751 9574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:05.412861 9574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:05.412969 9574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:05.413053 9574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:05.423154 9574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:05.424508 9574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:05.425863 9574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:05.438105 9579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:05.438071 9582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:05.438019 9580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:05.438964 9574 server_base.cc:1034] running on GCE node
I20260925 01:22:05.439733 9574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:05.441324 9574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:05.442628 9574 hybrid_clock.cc:648] HybridClock initialized: now 1790299325442530 us; error 86 us; skew 500 ppm
I20260925 01:22:05.443001 9574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:05 dist-test-slave-s12s krb5kdc[9561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:05.452862 9574 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:22:05.454496 9574 webserver.cc:460] Webserver started at http://127.0.0.1:34029/ using document root <none> and password file <none>
I20260925 01:22:05.455225 9574 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:05.455390 9574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:05.455710 9574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:05.458796 9574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "c20d25d8a31347fe98141e4a59c73315"
format_stamp: "Formatted at 2026-09-25 01:22:05 on dist-test-slave-s12s"
server_key: "dc61d605ddc5259f03f66ab4d02a7db0"
server_key_iv: "5bda58e89bff3ef64bdb806285980d8d"
server_key_version: "encryptionkey@0"
I20260925 01:22:05.459592 9574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "c20d25d8a31347fe98141e4a59c73315"
format_stamp: "Formatted at 2026-09-25 01:22:05 on dist-test-slave-s12s"
server_key: "dc61d605ddc5259f03f66ab4d02a7db0"
server_key_iv: "5bda58e89bff3ef64bdb806285980d8d"
server_key_version: "encryptionkey@0"
I20260925 01:22:05.464823 9574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:22:05.468252 9589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:05.469574 9574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:22:05.469770 9574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "c20d25d8a31347fe98141e4a59c73315"
format_stamp: "Formatted at 2026-09-25 01:22:05 on dist-test-slave-s12s"
server_key: "dc61d605ddc5259f03f66ab4d02a7db0"
server_key_iv: "5bda58e89bff3ef64bdb806285980d8d"
server_key_version: "encryptionkey@0"
I20260925 01:22:05.469973 9574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:05.491182 9574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:05.495380 9574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:05.495770 9574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:05.514941 9574 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44963
I20260925 01:22:05.514977 9640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44963 every 8 connection(s)
I20260925 01:22:05.516443 9574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:05.520453 9641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:05.526649 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 9574
I20260925 01:22:05.526853 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:05.527308 20890 external_mini_cluster.cc:1428] Setting key f64bfc2ff7ef0fb529dc409efa00579a
I20260925 01:22:05.531778 9641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315: Bootstrap starting.
I20260925 01:22:05.535881 9641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:05.537086 9641 log.cc:824] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:05.540376 9641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315: No bootstrap required, opened a new log
I20260925 01:22:05.542959 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:05.529417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:58473 (local address 127.0.0.1:44963)
0925 01:22:05.530211 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:05.530228 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:05.530282 (+ 54us) server_negotiation.cc:407] Connection header received
0925 01:22:05.532708 (+ 2426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:05.532736 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:05.533238 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:05.533732 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:05.534261 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:05.536261 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:05.537848 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:05.538700 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:05.538874 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:05.540122 (+ 1248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:05.540134 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:05.540152 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:05.540916 (+ 764us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:05.541014 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:05.541051 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:05.541223 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:05.541461 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 01:22:05.541635 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":314,"threads_started":1}
I20260925 01:22:05.546850 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:05.549064 9641 raft_consensus.cc:348] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20d25d8a31347fe98141e4a59c73315" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44963 } }
I20260925 01:22:05.549448 9641 raft_consensus.cc:374] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:22:05.549585 9641 raft_consensus.cc:729] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c20d25d8a31347fe98141e4a59c73315, State: Initialized, Role: FOLLOWER
I20260925 01:22:05.550257 9641 consensus_queue.cc:260] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20d25d8a31347fe98141e4a59c73315" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44963 } }
I20260925 01:22:05.550522 9641 raft_consensus.cc:388] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:22:05.550669 9641 raft_consensus.cc:482] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:22:05.550845 9641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:05.553583 9641 raft_consensus.cc:504] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20d25d8a31347fe98141e4a59c73315" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44963 } }
I20260925 01:22:05.554095 9641 leader_election.cc:302] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c20d25d8a31347fe98141e4a59c73315; no voters:
I20260925 01:22:05.554869 9641 leader_election.cc:288] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:22:05.554970 9651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:05.557641 9641 sys_catalog.cc:564] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:05.558457 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:05.551454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:40745 (local address 127.0.0.1:44963)
0925 01:22:05.551659 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:05.551667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:05.552052 (+ 385us) server_negotiation.cc:407] Connection header received
0925 01:22:05.552270 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:05.552276 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:05.552349 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:05.552466 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:05.553072 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:05.553856 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:05.555368 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:05.556088 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:05.556181 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:05.557652 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:05.557657 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:05.557660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:05.557678 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:05.557715 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:05.557730 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:05.557862 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:05.558113 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 01:22:05.558167 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 01:22:05.562036 9651 raft_consensus.cc:686] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [term 1 LEADER]: Becoming Leader. State: Replica: c20d25d8a31347fe98141e4a59c73315, State: Running, Role: LEADER
I20260925 01:22:05.562809 9651 consensus_queue.cc:237] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20d25d8a31347fe98141e4a59c73315" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44963 } }
I20260925 01:22:05.566279 9653 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c20d25d8a31347fe98141e4a59c73315. Latest consensus state: current_term: 1 leader_uuid: "c20d25d8a31347fe98141e4a59c73315" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20d25d8a31347fe98141e4a59c73315" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44963 } } }
I20260925 01:22:05.566793 9653 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:05.566270 9652 sys_catalog.cc:455] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c20d25d8a31347fe98141e4a59c73315" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c20d25d8a31347fe98141e4a59c73315" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44963 } } }
I20260925 01:22:05.568408 9652 sys_catalog.cc:458] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315 [sys.catalog]: This master's current role is: LEADER
W20260925 01:22:05.571969 9665 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:22:05.572106 9665 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:22:05.572326 9666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:22:05.574345 9666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:05.582898 9666 catalog_manager.cc:1347] Generated new cluster ID: f873141379d9483188fb8f6fb95c62ff
I20260925 01:22:05.583065 9666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:05.601361 9666 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:05.602679 9666 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:05.611588 9666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c20d25d8a31347fe98141e4a59c73315: Generated new TSK 0
I20260925 01:22:05.612411 9666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:22:05.625228 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9574
W20260925 01:22:05.626742 9636 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.63:58473
W20260925 01:22:05.627688 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9574: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:22:05.627843 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 9574
W20260925 01:22:05.647060 9559 connection.cc:535] client connection to 127.0.0.1:44963 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44963): Connection reset by peer (error 104)
2026-09-25T01:22:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1933 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:05 dist-test-slave-s12s krb5kdc[9672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:05 dist-test-slave-s12s krb5kdc[9672](info): set up 2 sockets
Sep 25 01:22:05 dist-test-slave-s12s krb5kdc[9672](info): commencing operation
krb5kdc: starting...
W20260925 01:22:06.753289 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:06 dist-test-slave-s12s krb5kdc[9672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:22:07.022297 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:07.358409 9685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:07.358708 9685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:07.358916 9685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:07.366156 9685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:07.366264 9685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:07.366384 9685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:07.366499 9685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:07.366581 9685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:07.376451 9685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:07.377848 9685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:07.379252 9685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:07.391803 9693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:07.391831 9691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:07.392040 9690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:07.392529 9685 server_base.cc:1034] running on GCE node
I20260925 01:22:07.393399 9685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:07.394939 9685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:07.396160 9685 hybrid_clock.cc:648] HybridClock initialized: now 1790299327396081 us; error 67 us; skew 500 ppm
I20260925 01:22:07.396598 9685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:07 dist-test-slave-s12s krb5kdc[9672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:07.399842 9685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:22:07.401530 9685 webserver.cc:460] Webserver started at http://127.0.0.1:42611/ using document root <none> and password file <none>
I20260925 01:22:07.402220 9685 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:07.402328 9685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:07.402599 9685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:07.405719 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "c077c0cf65574a7ca0d606d11aaa0a1a"
format_stamp: "Formatted at 2026-09-25 01:22:07 on dist-test-slave-s12s"
server_key: "80df960381058f2ebce8b59be7e918bc"
server_key_iv: "1bd95f600774586ebe7c60d7d0f0c124"
server_key_version: "encryptionkey@0"
I20260925 01:22:07.406455 9685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "c077c0cf65574a7ca0d606d11aaa0a1a"
format_stamp: "Formatted at 2026-09-25 01:22:07 on dist-test-slave-s12s"
server_key: "80df960381058f2ebce8b59be7e918bc"
server_key_iv: "1bd95f600774586ebe7c60d7d0f0c124"
server_key_version: "encryptionkey@0"
I20260925 01:22:07.411835 9685 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:22:07.415515 9700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:07.417261 9685 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:22:07.417462 9685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "c077c0cf65574a7ca0d606d11aaa0a1a"
format_stamp: "Formatted at 2026-09-25 01:22:07 on dist-test-slave-s12s"
server_key: "80df960381058f2ebce8b59be7e918bc"
server_key_iv: "1bd95f600774586ebe7c60d7d0f0c124"
server_key_version: "encryptionkey@0"
I20260925 01:22:07.417707 9685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:07.424177 9685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:07.428440 9685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:07.428848 9685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:07.447803 9685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41279
I20260925 01:22:07.447850 9751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41279 every 8 connection(s)
I20260925 01:22:07.449265 9685 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:07.453037 9752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:07.457850 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 9685
I20260925 01:22:07.458027 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:07.458459 20890 external_mini_cluster.cc:1428] Setting key aaf5bc29ab2fa50496c29fb1cdc33296
I20260925 01:22:07.463811 9752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a: Bootstrap starting.
I20260925 01:22:07.466269 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:07.460328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:41597 (local address 127.0.0.1:41279)
0925 01:22:07.461140 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:07.461154 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:07.461194 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:22:07.462824 (+ 1630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:07.462847 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:07.463176 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:07.463480 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:07.464618 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:07.464625 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:07.464637 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:07.464652 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:07.464765 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:07.464788 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:07.464949 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:07.465145 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 01:22:07.465350 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":374,"threads_started":1}
I20260925 01:22:07.467367 9752 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:07.468618 9752 log.cc:824] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:07.469919 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:07.471517 9752 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a: No bootstrap required, opened a new log
I20260925 01:22:07.477015 9755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:07.474336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:49913 (local address 127.0.0.1:41279)
0925 01:22:07.474527 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:07.474534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:07.475084 (+ 550us) server_negotiation.cc:407] Connection header received
0925 01:22:07.475206 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:07.475211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:07.475280 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:07.475379 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:07.476298 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:07.476304 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:07.476309 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:07.476326 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:07.476378 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:07.476401 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:07.476507 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:07.476615 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 01:22:07.476706 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 01:22:07.479825 9752 raft_consensus.cc:348] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c077c0cf65574a7ca0d606d11aaa0a1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } }
I20260925 01:22:07.480245 9752 raft_consensus.cc:374] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:22:07.480368 9752 raft_consensus.cc:729] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c077c0cf65574a7ca0d606d11aaa0a1a, State: Initialized, Role: FOLLOWER
I20260925 01:22:07.481009 9752 consensus_queue.cc:260] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c077c0cf65574a7ca0d606d11aaa0a1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } }
I20260925 01:22:07.481269 9752 raft_consensus.cc:388] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:22:07.481412 9752 raft_consensus.cc:482] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:22:07.481595 9752 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:07.483779 9752 raft_consensus.cc:504] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c077c0cf65574a7ca0d606d11aaa0a1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } }
I20260925 01:22:07.484292 9752 leader_election.cc:302] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c077c0cf65574a7ca0d606d11aaa0a1a; no voters:
I20260925 01:22:07.484928 9752 leader_election.cc:288] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:22:07.485086 9762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:07.485843 9762 raft_consensus.cc:686] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [term 1 LEADER]: Becoming Leader. State: Replica: c077c0cf65574a7ca0d606d11aaa0a1a, State: Running, Role: LEADER
I20260925 01:22:07.486413 9762 consensus_queue.cc:237] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c077c0cf65574a7ca0d606d11aaa0a1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } }
I20260925 01:22:07.487100 9752 sys_catalog.cc:564] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:07.489842 9763 sys_catalog.cc:455] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c077c0cf65574a7ca0d606d11aaa0a1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c077c0cf65574a7ca0d606d11aaa0a1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } } }
I20260925 01:22:07.490401 9763 sys_catalog.cc:458] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:07.489868 9764 sys_catalog.cc:455] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c077c0cf65574a7ca0d606d11aaa0a1a. Latest consensus state: current_term: 1 leader_uuid: "c077c0cf65574a7ca0d606d11aaa0a1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c077c0cf65574a7ca0d606d11aaa0a1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41279 } } }
I20260925 01:22:07.491269 9764 sys_catalog.cc:458] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:07.493618 9767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:22:07.499096 9767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:07.508165 9767 catalog_manager.cc:1347] Generated new cluster ID: 4f3da209513f467b8c699388f6a6439f
I20260925 01:22:07.508337 9767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:07.521097 9767 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:07.521606 9767 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:07.530977 9767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c077c0cf65574a7ca0d606d11aaa0a1a: Generated new TSK 0
I20260925 01:22:07.532029 9767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:22:07.543440 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9685
W20260925 01:22:07.545318 9747 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.63:41597
W20260925 01:22:07.546088 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9685: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:22:07.546257 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 9685
2026-09-25T01:22:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1916 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:07 dist-test-slave-s12s krb5kdc[9783](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:07 dist-test-slave-s12s krb5kdc[9783](info): set up 2 sockets
Sep 25 01:22:07 dist-test-slave-s12s krb5kdc[9783](info): commencing operation
krb5kdc: starting...
W20260925 01:22:08.649204 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:08 dist-test-slave-s12s krb5kdc[9783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:22:08.911713 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:09.250633 9796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:09.250926 9796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:09.251109 9796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:09.258589 9796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:09.258699 9796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:09.258813 9796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:09.258919 9796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:09.259001 9796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:09.269588 9796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:09.270933 9796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:09.272804 9796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:09.285189 9804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:09.285312 9802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:09.285347 9801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:09.286479 9796 server_base.cc:1034] running on GCE node
I20260925 01:22:09.287390 9796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:09.289222 9796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:09.290441 9796 hybrid_clock.cc:648] HybridClock initialized: now 1790299329290399 us; error 31 us; skew 500 ppm
I20260925 01:22:09.290843 9796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:09 dist-test-slave-s12s krb5kdc[9783](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:09.294188 9796 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:22:09.295802 9796 webserver.cc:460] Webserver started at http://127.0.0.1:45669/ using document root <none> and password file <none>
I20260925 01:22:09.296520 9796 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:09.296717 9796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:09.297050 9796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:09.300055 9796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "581ebb70da574d068d1b5b89e7cc6ad7"
format_stamp: "Formatted at 2026-09-25 01:22:09 on dist-test-slave-s12s"
server_key: "55cd199b2a701053770638da3dceac16"
server_key_iv: "7195739264cc926cf9e734a678e34e3d"
server_key_version: "encryptionkey@0"
I20260925 01:22:09.300916 9796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "581ebb70da574d068d1b5b89e7cc6ad7"
format_stamp: "Formatted at 2026-09-25 01:22:09 on dist-test-slave-s12s"
server_key: "55cd199b2a701053770638da3dceac16"
server_key_iv: "7195739264cc926cf9e734a678e34e3d"
server_key_version: "encryptionkey@0"
I20260925 01:22:09.306289 9796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:22:09.309839 9811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:09.311170 9796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:22:09.311363 9796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "581ebb70da574d068d1b5b89e7cc6ad7"
format_stamp: "Formatted at 2026-09-25 01:22:09 on dist-test-slave-s12s"
server_key: "55cd199b2a701053770638da3dceac16"
server_key_iv: "7195739264cc926cf9e734a678e34e3d"
server_key_version: "encryptionkey@0"
I20260925 01:22:09.311558 9796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:09.325441 9796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:09.329711 9796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:09.330057 9796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:09.350526 9796 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38885
I20260925 01:22:09.350557 9862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38885 every 8 connection(s)
I20260925 01:22:09.351924 9796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:09.355758 9863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:09.357398 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 9796
I20260925 01:22:09.357616 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:09.358063 20890 external_mini_cluster.cc:1428] Setting key 7fe733b1005a3a795d2c12f017e4863c
I20260925 01:22:09.369668 9863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7: Bootstrap starting.
Sep 25 01:22:09 dist-test-slave-s12s krb5kdc[9783](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 01:22:09.373088 9863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:09.374343 9863 log.cc:824] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:09.377300 9863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7: No bootstrap required, opened a new log
I20260925 01:22:09.380885 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:09.359993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:34965 (local address 127.0.0.1:38885)
0925 01:22:09.360891 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:09.360903 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:09.360952 (+ 49us) server_negotiation.cc:407] Connection header received
0925 01:22:09.362663 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:09.362691 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:09.363108 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:09.363506 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:09.365040 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:09.366336 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:09.367746 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:09.368452 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:09.368666 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:09.373290 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:09.373299 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:09.373314 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:09.373358 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:09.376523 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:09.377374 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:09.377380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:09.377387 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:09.377593 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:09.378079 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:09.378082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:09.378084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:09.378601 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:09.379052 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:09.379492 (+ 440us) server_negotiation.cc:299] Negotiation successful
0925 01:22:09.379813 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":396,"threads_started":1}
I20260925 01:22:09.384976 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:09.386363 9863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581ebb70da574d068d1b5b89e7cc6ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260925 01:22:09.386739 9863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:22:09.386845 9863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 581ebb70da574d068d1b5b89e7cc6ad7, State: Initialized, Role: FOLLOWER
I20260925 01:22:09.387400 9863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581ebb70da574d068d1b5b89e7cc6ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260925 01:22:09.387640 9863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:22:09.387809 9863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:22:09.388007 9863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:09.390762 9863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581ebb70da574d068d1b5b89e7cc6ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260925 01:22:09.391299 9863 leader_election.cc:302] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 581ebb70da574d068d1b5b89e7cc6ad7; no voters:
I20260925 01:22:09.391928 9863 leader_election.cc:288] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:22:09.392071 9873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:09.392819 9873 raft_consensus.cc:686] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [term 1 LEADER]: Becoming Leader. State: Replica: 581ebb70da574d068d1b5b89e7cc6ad7, State: Running, Role: LEADER
I20260925 01:22:09.393527 9873 consensus_queue.cc:237] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581ebb70da574d068d1b5b89e7cc6ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } }
I20260925 01:22:09.395545 9873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 581ebb70da574d068d1b5b89e7cc6ad7. Latest consensus state: current_term: 1 leader_uuid: "581ebb70da574d068d1b5b89e7cc6ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581ebb70da574d068d1b5b89e7cc6ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } } }
I20260925 01:22:09.397148 9873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:09.396775 9875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "581ebb70da574d068d1b5b89e7cc6ad7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "581ebb70da574d068d1b5b89e7cc6ad7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38885 } } }
I20260925 01:22:09.397558 9875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:09.399839 9863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:09.403657 9877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:22:09.410964 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:09.389671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:58821 (local address 127.0.0.1:38885)
0925 01:22:09.389900 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:09.389909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:09.390443 (+ 534us) server_negotiation.cc:407] Connection header received
0925 01:22:09.390559 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:09.390566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:09.390651 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:09.390762 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:09.392806 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:09.393998 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:09.395880 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:09.396672 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:09.396819 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:09.400503 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:09.400510 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:09.400566 (+ 56us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:09.400610 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:09.405894 (+ 5284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:09.407259 (+ 1365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:09.407268 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:09.407275 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:09.407456 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:09.408174 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:09.408182 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:09.408188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:09.408790 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:09.409109 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:09.410380 (+ 1271us) server_negotiation.cc:299] Negotiation successful
0925 01:22:09.410585 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 01:22:09.412667 9877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:09.424854 9877 catalog_manager.cc:1347] Generated new cluster ID: 5e95ccaa126249499c97d1638088e652
I20260925 01:22:09.425004 9877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:09.433290 9877 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:09.434580 9877 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:09.449102 9877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 581ebb70da574d068d1b5b89e7cc6ad7: Generated new TSK 0
I20260925 01:22:09.450273 9877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:22:09.464290 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9796
I20260925 01:22:09.465602 9858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:22:09.555955 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 9796
W20260925 01:22:09.574225 9781 connection.cc:535] client connection to 127.0.0.1:38885 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38885): Connection reset by peer (error 104)
2026-09-25T01:22:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2010 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:09 dist-test-slave-s12s krb5kdc[9895](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:09 dist-test-slave-s12s krb5kdc[9895](info): set up 2 sockets
Sep 25 01:22:09 dist-test-slave-s12s krb5kdc[9895](info): commencing operation
krb5kdc: starting...
W20260925 01:22:10.665282 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:10 dist-test-slave-s12s krb5kdc[9895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:22:10.930143 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:11.271271 9908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:11.271584 9908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:11.271679 9908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:11.279203 9908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:11.279326 9908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:11.279374 9908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:11.279438 9908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:11.279484 9908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:11.289491 9908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33861
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:11.290812 9908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:11.292152 9908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:11.304399 9916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:11.304450 9914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:11.304539 9913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:11.305632 9908 server_base.cc:1034] running on GCE node
I20260925 01:22:11.306509 9908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:11.307875 9908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:11.309077 9908 hybrid_clock.cc:648] HybridClock initialized: now 1790299331309033 us; error 31 us; skew 500 ppm
I20260925 01:22:11.309449 9908 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:11 dist-test-slave-s12s krb5kdc[9895](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:11.318393 9908 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:22:11.320004 9908 webserver.cc:460] Webserver started at http://127.0.0.1:37509/ using document root <none> and password file <none>
I20260925 01:22:11.320820 9908 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:11.320956 9908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:11.321291 9908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:11.324389 9908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "1a1269270f6e4791ad3bf2f54654123c"
format_stamp: "Formatted at 2026-09-25 01:22:11 on dist-test-slave-s12s"
server_key: "7b95c859d324cfbce1de6c2d9446af53"
server_key_iv: "d41203d20cbc23870c193a8db6944242"
server_key_version: "encryptionkey@0"
I20260925 01:22:11.325273 9908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "1a1269270f6e4791ad3bf2f54654123c"
format_stamp: "Formatted at 2026-09-25 01:22:11 on dist-test-slave-s12s"
server_key: "7b95c859d324cfbce1de6c2d9446af53"
server_key_iv: "d41203d20cbc23870c193a8db6944242"
server_key_version: "encryptionkey@0"
I20260925 01:22:11.330639 9908 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:22:11.334159 9923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:11.335536 9908 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:22:11.335733 9908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "1a1269270f6e4791ad3bf2f54654123c"
format_stamp: "Formatted at 2026-09-25 01:22:11 on dist-test-slave-s12s"
server_key: "7b95c859d324cfbce1de6c2d9446af53"
server_key_iv: "d41203d20cbc23870c193a8db6944242"
server_key_version: "encryptionkey@0"
I20260925 01:22:11.335935 9908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:11.370609 9908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:11.375048 9908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:11.375375 9908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:11.394591 9908 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33861
I20260925 01:22:11.394654 9974 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33861 every 8 connection(s)
I20260925 01:22:11.396144 9908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:11.398571 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 9908
I20260925 01:22:11.398778 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:11.399262 20890 external_mini_cluster.cc:1428] Setting key 51bfe273f90ee596cbf44607be6c8579
I20260925 01:22:11.400357 9975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:11.413146 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:11.401134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:55987 (local address 127.0.0.1:33861)
0925 01:22:11.401940 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:11.401952 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:11.401992 (+ 40us) server_negotiation.cc:407] Connection header received
0925 01:22:11.403582 (+ 1590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:11.403605 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:11.404011 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:11.404355 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:11.404984 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:11.406356 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:11.408237 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:11.409115 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:11.409252 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:11.410693 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:11.410701 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:11.410717 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:11.411289 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:11.411371 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:11.411402 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:11.411530 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:11.411817 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 01:22:11.412056 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":381,"threads_started":1}
I20260925 01:22:11.414306 9975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c: Bootstrap starting.
I20260925 01:22:11.417250 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:11.417853 9975 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:11.418856 9975 log.cc:824] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:11.422245 9975 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c: No bootstrap required, opened a new log
I20260925 01:22:11.428095 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:11.421735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:57759 (local address 127.0.0.1:33861)
0925 01:22:11.421947 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:11.421955 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:11.422375 (+ 420us) server_negotiation.cc:407] Connection header received
0925 01:22:11.422606 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:11.422610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:11.422674 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:11.422797 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:11.423319 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:11.424189 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:11.425620 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:11.426086 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:11.426228 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:11.427373 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:11.427378 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:11.427381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:11.427398 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:11.427435 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:11.427450 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:11.427584 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:11.427776 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 01:22:11.427838 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:22:11.430750 9975 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1269270f6e4791ad3bf2f54654123c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33861 } }
I20260925 01:22:11.431137 9975 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:22:11.431262 9975 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1269270f6e4791ad3bf2f54654123c, State: Initialized, Role: FOLLOWER
I20260925 01:22:11.431830 9975 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1269270f6e4791ad3bf2f54654123c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33861 } }
I20260925 01:22:11.432076 9975 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:22:11.432226 9975 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:22:11.432401 9975 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:11.434798 9975 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1269270f6e4791ad3bf2f54654123c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33861 } }
I20260925 01:22:11.435415 9975 leader_election.cc:302] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a1269270f6e4791ad3bf2f54654123c; no voters:
I20260925 01:22:11.436112 9975 leader_election.cc:288] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:22:11.436272 9985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:11.437100 9985 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [term 1 LEADER]: Becoming Leader. State: Replica: 1a1269270f6e4791ad3bf2f54654123c, State: Running, Role: LEADER
I20260925 01:22:11.437672 9985 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1269270f6e4791ad3bf2f54654123c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33861 } }
I20260925 01:22:11.438364 9975 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:11.440455 9986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a1269270f6e4791ad3bf2f54654123c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1269270f6e4791ad3bf2f54654123c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33861 } } }
I20260925 01:22:11.440825 9986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:11.441208 9987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a1269270f6e4791ad3bf2f54654123c. Latest consensus state: current_term: 1 leader_uuid: "1a1269270f6e4791ad3bf2f54654123c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1269270f6e4791ad3bf2f54654123c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33861 } } }
I20260925 01:22:11.441612 9987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:11.444602 9991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:22:11.450451 9991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:11.459750 9991 catalog_manager.cc:1347] Generated new cluster ID: a2a19b14acad4f8d8e864bc4a08f580b
I20260925 01:22:11.459872 9991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:11.468343 9991 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:11.470161 9991 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:11.487073 9991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a1269270f6e4791ad3bf2f54654123c: Generated new TSK 0
I20260925 01:22:11.487972 9991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:22:11.499104 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9908
W20260925 01:22:11.500203 9970 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.63:55987
W20260925 01:22:11.500984 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 9908: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:22:11.501142 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 9908
W20260925 01:22:11.518802 9893 connection.cc:535] client connection to 127.0.0.1:33861 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33861): Connection reset by peer (error 104)
2026-09-25T01:22:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1944 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:11 dist-test-slave-s12s krb5kdc[10006](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:11 dist-test-slave-s12s krb5kdc[10006](info): set up 2 sockets
Sep 25 01:22:11 dist-test-slave-s12s krb5kdc[10006](info): commencing operation
krb5kdc: starting...
W20260925 01:22:12.621313 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:12 dist-test-slave-s12s krb5kdc[10006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 01:22:12.881446 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39847
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:13.220541 10019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:13.220942 10019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:13.221112 10019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:13.228461 10019 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:13.228567 10019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:13.228704 10019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:13.228818 10019 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:13.228899 10019 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:13.238966 10019 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39847
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:13.240324 10019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:13.241700 10019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:13.253959 10027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:13.254089 10024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:13.254078 10025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:13.255075 10019 server_base.cc:1034] running on GCE node
I20260925 01:22:13.256055 10019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:13.257599 10019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:13.258837 10019 hybrid_clock.cc:648] HybridClock initialized: now 1790299333258798 us; error 59 us; skew 500 ppm
I20260925 01:22:13.259222 10019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:13 dist-test-slave-s12s krb5kdc[10006](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:13.262532 10019 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 01:22:13.264019 10019 webserver.cc:460] Webserver started at http://127.0.0.1:32837/ using document root <none> and password file <none>
I20260925 01:22:13.264663 10019 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:13.264833 10019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:13.265164 10019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:13.268159 10019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "6183ef338c8945a2ac6ee84b801a8081"
format_stamp: "Formatted at 2026-09-25 01:22:13 on dist-test-slave-s12s"
server_key: "471fd3c59cadbd9d022dc7a67a71eaea"
server_key_iv: "76f495d7514d858ed39455a05ed38359"
server_key_version: "encryptionkey@0"
I20260925 01:22:13.268962 10019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "6183ef338c8945a2ac6ee84b801a8081"
format_stamp: "Formatted at 2026-09-25 01:22:13 on dist-test-slave-s12s"
server_key: "471fd3c59cadbd9d022dc7a67a71eaea"
server_key_iv: "76f495d7514d858ed39455a05ed38359"
server_key_version: "encryptionkey@0"
I20260925 01:22:13.274009 10019 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260925 01:22:13.277593 10034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:13.279054 10019 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 01:22:13.279268 10019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "6183ef338c8945a2ac6ee84b801a8081"
format_stamp: "Formatted at 2026-09-25 01:22:13 on dist-test-slave-s12s"
server_key: "471fd3c59cadbd9d022dc7a67a71eaea"
server_key_iv: "76f495d7514d858ed39455a05ed38359"
server_key_version: "encryptionkey@0"
I20260925 01:22:13.279486 10019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:13.285893 10019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:13.290381 10019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:13.290730 10019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:13.310935 10019 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39847
I20260925 01:22:13.310981 10085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39847 every 8 connection(s)
I20260925 01:22:13.312476 10019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:13.316395 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 10019
I20260925 01:22:13.316591 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:13.316593 10086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:13.317183 20890 external_mini_cluster.cc:1428] Setting key 6d35f9efb68797b72807ed8c505bc0c0
I20260925 01:22:13.325012 10089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:13.319005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:35039 (local address 127.0.0.1:39847)
0925 01:22:13.319801 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:13.319812 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:13.319854 (+ 42us) server_negotiation.cc:407] Connection header received
0925 01:22:13.321543 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:13.321572 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:13.321908 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:13.322326 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:13.323378 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:13.323383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:13.323397 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:13.323419 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:13.323481 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:13.323500 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:13.323755 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:13.323895 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 01:22:13.324071 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":325,"threads_started":1}
I20260925 01:22:13.328637 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:13.329686 10086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081: Bootstrap starting.
I20260925 01:22:13.332948 10086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:13.334174 10086 log.cc:824] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:13.335827 10089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:13.333066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:41459 (local address 127.0.0.1:39847)
0925 01:22:13.333226 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:13.333230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:13.333702 (+ 472us) server_negotiation.cc:407] Connection header received
0925 01:22:13.333903 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:13.333910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:13.333954 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:13.334062 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:13.335121 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:13.335124 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:13.335127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:13.335141 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:13.335177 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:13.335191 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:13.335300 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:13.335449 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 01:22:13.335492 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 01:22:13.337469 10086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081: No bootstrap required, opened a new log
I20260925 01:22:13.344519 10086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6183ef338c8945a2ac6ee84b801a8081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39847 } }
I20260925 01:22:13.344935 10086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:22:13.345067 10086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6183ef338c8945a2ac6ee84b801a8081, State: Initialized, Role: FOLLOWER
I20260925 01:22:13.345613 10086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6183ef338c8945a2ac6ee84b801a8081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39847 } }
I20260925 01:22:13.345844 10086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:22:13.345993 10086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:22:13.346218 10086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:13.348368 10086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6183ef338c8945a2ac6ee84b801a8081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39847 } }
I20260925 01:22:13.348870 10086 leader_election.cc:302] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6183ef338c8945a2ac6ee84b801a8081; no voters:
I20260925 01:22:13.349442 10086 leader_election.cc:288] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:22:13.349623 10096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:13.350270 10096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [term 1 LEADER]: Becoming Leader. State: Replica: 6183ef338c8945a2ac6ee84b801a8081, State: Running, Role: LEADER
I20260925 01:22:13.350800 10096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6183ef338c8945a2ac6ee84b801a8081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39847 } }
I20260925 01:22:13.351507 10086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:13.354549 10097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6183ef338c8945a2ac6ee84b801a8081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6183ef338c8945a2ac6ee84b801a8081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39847 } } }
I20260925 01:22:13.355073 10097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:13.355618 10098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6183ef338c8945a2ac6ee84b801a8081. Latest consensus state: current_term: 1 leader_uuid: "6183ef338c8945a2ac6ee84b801a8081" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6183ef338c8945a2ac6ee84b801a8081" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39847 } } }
I20260925 01:22:13.356009 10098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081 [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:13.356909 10105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:22:13.361186 10105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:13.372668 10105 catalog_manager.cc:1347] Generated new cluster ID: 11cbf8d787704605b173e9ab77caef34
I20260925 01:22:13.372814 10105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:13.385732 10105 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:13.386157 10105 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:13.401223 10105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6183ef338c8945a2ac6ee84b801a8081: Generated new TSK 0
I20260925 01:22:13.402159 10105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:22:13.419111 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10019
W20260925 01:22:13.421074 10081 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.63:35039
W20260925 01:22:13.421792 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10019: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:22:13.426312 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 10019
2026-09-25T01:22:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1925 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:13 dist-test-slave-s12s krb5kdc[10117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:13 dist-test-slave-s12s krb5kdc[10117](info): set up 2 sockets
Sep 25 01:22:13 dist-test-slave-s12s krb5kdc[10117](info): commencing operation
krb5kdc: starting...
W20260925 01:22:14.549350 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:14 dist-test-slave-s12s krb5kdc[10117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:22:14.809944 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:40737
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:34107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:40737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:15.147996 10130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:15.148348 10130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:15.148516 10130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:15.155740 10130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:15.155889 10130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:15.155995 10130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:15.156113 10130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:15.156199 10130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:15.166023 10130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:34107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:40737
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:40737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:15.167393 10130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:15.168812 10130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:15.180737 10138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:15.180799 10135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:15.180776 10136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:15.181517 10130 server_base.cc:1034] running on GCE node
I20260925 01:22:15.182246 10130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:15.183614 10130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:15.184871 10130 hybrid_clock.cc:648] HybridClock initialized: now 1790299335184826 us; error 56 us; skew 500 ppm
I20260925 01:22:15.185228 10130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:15 dist-test-slave-s12s krb5kdc[10117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299335, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:15.188297 10130 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:22:15.189900 10130 webserver.cc:460] Webserver started at http://127.20.102.190:46579/ using document root <none> and password file <none>
I20260925 01:22:15.190596 10130 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:15.190743 10130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:15.191054 10130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:15.194087 10130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "f0f4a4384a144c4f984a99fbd185be63"
format_stamp: "Formatted at 2026-09-25 01:22:15 on dist-test-slave-s12s"
server_key: "d8f97c152f97a1ec382e854a73857015"
server_key_iv: "037fe8dceaae6665f82ec06082974406"
server_key_version: "encryptionkey@0"
I20260925 01:22:15.194870 10130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "f0f4a4384a144c4f984a99fbd185be63"
format_stamp: "Formatted at 2026-09-25 01:22:15 on dist-test-slave-s12s"
server_key: "d8f97c152f97a1ec382e854a73857015"
server_key_iv: "037fe8dceaae6665f82ec06082974406"
server_key_version: "encryptionkey@0"
I20260925 01:22:15.199940 10130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 01:22:15.203388 10145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:15.204771 10130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 01:22:15.204944 10130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "f0f4a4384a144c4f984a99fbd185be63"
format_stamp: "Formatted at 2026-09-25 01:22:15 on dist-test-slave-s12s"
server_key: "d8f97c152f97a1ec382e854a73857015"
server_key_iv: "037fe8dceaae6665f82ec06082974406"
server_key_version: "encryptionkey@0"
I20260925 01:22:15.205140 10130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:15.228056 10130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:15.232239 10130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:15.232538 10130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:15.251542 10130 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:40737
I20260925 01:22:15.251573 10196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:40737 every 8 connection(s)
I20260925 01:22:15.252949 10130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:15.255617 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 10130
I20260925 01:22:15.255808 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:15.256340 20890 external_mini_cluster.cc:1428] Setting key f2d3563f05bd8bc61204af6059af5a3f
I20260925 01:22:15.257251 10197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 01:22:15 dist-test-slave-s12s krb5kdc[10117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:22:15.269819 10197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63: Bootstrap starting.
I20260925 01:22:15.273860 10197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:15.275300 10197 log.cc:824] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:15.278867 10197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63: No bootstrap required, opened a new log
I20260925 01:22:15.280218 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:15.258166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49997 (local address 127.20.102.190:40737)
0925 01:22:15.258905 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:15.258916 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:15.258954 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:22:15.260603 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:15.260625 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:15.261082 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:15.261492 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:15.263050 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:15.264398 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:15.265720 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:15.266397 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:15.266603 (+ 206us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:15.271044 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:15.271055 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:15.271074 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:15.271126 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:15.275341 (+ 4215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:15.276310 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:15.276317 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:15.276323 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:15.276496 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:15.277024 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:15.277029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:15.277032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:15.277778 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:15.278125 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:15.278474 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 01:22:15.278815 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":340,"threads_started":1}
I20260925 01:22:15.284487 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:15.287772 10197 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f4a4384a144c4f984a99fbd185be63" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 40737 } }
I20260925 01:22:15.288175 10197 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:22:15.288282 10197 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f4a4384a144c4f984a99fbd185be63, State: Initialized, Role: FOLLOWER
I20260925 01:22:15.288959 10197 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f4a4384a144c4f984a99fbd185be63" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 40737 } }
I20260925 01:22:15.289292 10197 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:22:15.289449 10197 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:22:15.289626 10197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:15.292313 10197 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f4a4384a144c4f984a99fbd185be63" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 40737 } }
I20260925 01:22:15.292891 10197 leader_election.cc:302] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0f4a4384a144c4f984a99fbd185be63; no voters:
I20260925 01:22:15.293625 10197 leader_election.cc:288] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:22:15.293730 10207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:15.295019 10207 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [term 1 LEADER]: Becoming Leader. State: Replica: f0f4a4384a144c4f984a99fbd185be63, State: Running, Role: LEADER
I20260925 01:22:15.295677 10207 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f4a4384a144c4f984a99fbd185be63" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 40737 } }
I20260925 01:22:15.296126 10197 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:15.298887 10209 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0f4a4384a144c4f984a99fbd185be63. Latest consensus state: current_term: 1 leader_uuid: "f0f4a4384a144c4f984a99fbd185be63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f4a4384a144c4f984a99fbd185be63" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 40737 } } }
I20260925 01:22:15.298923 10208 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0f4a4384a144c4f984a99fbd185be63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f4a4384a144c4f984a99fbd185be63" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 40737 } } }
I20260925 01:22:15.299410 10209 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:15.299404 10208 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63 [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:15.303158 10213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:22:15.308302 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:15.289009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57799 (local address 127.20.102.190:40737)
0925 01:22:15.289200 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:15.289208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:15.289671 (+ 463us) server_negotiation.cc:407] Connection header received
0925 01:22:15.289846 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:15.289851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:15.289939 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:15.290044 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:15.291426 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:15.292539 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:15.294177 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:15.294914 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:15.295050 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:15.298942 (+ 3892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:15.298951 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:15.298956 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:15.299004 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:15.303791 (+ 4787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:15.304929 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:15.304947 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:15.304950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:15.305192 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:15.305975 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:15.305980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:15.305983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:15.306453 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:15.306776 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:15.307582 (+ 806us) server_negotiation.cc:299] Negotiation successful
0925 01:22:15.307800 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:22:15.310068 10213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:15.319850 10213 catalog_manager.cc:1347] Generated new cluster ID: 0e042022e51d4e8d9a5eba1c1257eccc
I20260925 01:22:15.320005 10213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:15.340901 10213 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:15.342267 10213 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:15.354748 10213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0f4a4384a144c4f984a99fbd185be63: Generated new TSK 0
I20260925 01:22:15.355605 10213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:22:15.368759 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10130
I20260925 01:22:15.370399 10192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:22:15.462535 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 10130
W20260925 01:22:15.480906 10115 connection.cc:535] client connection to 127.20.102.190:40737 recv error: Network error: failed to read from TLS socket (remote: 127.20.102.190:40737): Connection reset by peer (error 104)
2026-09-25T01:22:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2036 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:15 dist-test-slave-s12s krb5kdc[10229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:15 dist-test-slave-s12s krb5kdc[10229](info): set up 2 sockets
Sep 25 01:22:15 dist-test-slave-s12s krb5kdc[10229](info): commencing operation
krb5kdc: starting...
W20260925 01:22:16.589143 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:16 dist-test-slave-s12s krb5kdc[10229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:22:16.849998 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39325
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:39561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:39325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:17.187654 10242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:17.187950 10242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:17.188123 10242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:17.195274 10242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:17.195382 10242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:17.195492 10242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:17.195619 10242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:17.195683 10242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:17.205585 10242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:39325
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:17.206913 10242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:17.208269 10242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:17.220345 10250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:17.220407 10247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:17.220430 10248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:17.221681 10242 server_base.cc:1034] running on GCE node
I20260925 01:22:17.222509 10242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:17.223896 10242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:17.225121 10242 hybrid_clock.cc:648] HybridClock initialized: now 1790299337225078 us; error 55 us; skew 500 ppm
I20260925 01:22:17.225464 10242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:17 dist-test-slave-s12s krb5kdc[10229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299337, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:17.228525 10242 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:22:17.230080 10242 webserver.cc:460] Webserver started at http://127.20.102.190:39989/ using document root <none> and password file <none>
I20260925 01:22:17.230713 10242 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:17.230814 10242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:17.231065 10242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:17.234014 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "bbd16e4c9a2541a9b7868b5d4752bd8d"
format_stamp: "Formatted at 2026-09-25 01:22:17 on dist-test-slave-s12s"
server_key: "8092067cc9baec2763527dad475da548"
server_key_iv: "4ab28372a60f2ed6f75de3463d2c1be9"
server_key_version: "encryptionkey@0"
I20260925 01:22:17.234717 10242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "bbd16e4c9a2541a9b7868b5d4752bd8d"
format_stamp: "Formatted at 2026-09-25 01:22:17 on dist-test-slave-s12s"
server_key: "8092067cc9baec2763527dad475da548"
server_key_iv: "4ab28372a60f2ed6f75de3463d2c1be9"
server_key_version: "encryptionkey@0"
I20260925 01:22:17.239727 10242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 01:22:17.243113 10257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:17.244465 10242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 01:22:17.244638 10242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "bbd16e4c9a2541a9b7868b5d4752bd8d"
format_stamp: "Formatted at 2026-09-25 01:22:17 on dist-test-slave-s12s"
server_key: "8092067cc9baec2763527dad475da548"
server_key_iv: "4ab28372a60f2ed6f75de3463d2c1be9"
server_key_version: "encryptionkey@0"
I20260925 01:22:17.244879 10242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:17.263451 10242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:17.267443 10242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:17.267752 10242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:17.288760 10242 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:39325
I20260925 01:22:17.288822 10308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:39325 every 8 connection(s)
I20260925 01:22:17.290242 10242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:17.294186 10309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:17.294595 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 10242
I20260925 01:22:17.294797 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:17.295374 20890 external_mini_cluster.cc:1428] Setting key aab82c56e390c60d497857876d778f62
I20260925 01:22:17.307971 10309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d: Bootstrap starting.
I20260925 01:22:17.309656 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:17.297230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53965 (local address 127.20.102.190:39325)
0925 01:22:17.298076 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:17.298086 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:17.298136 (+ 50us) server_negotiation.cc:407] Connection header received
0925 01:22:17.300443 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:17.300469 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:17.300930 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:17.301273 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:17.301904 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:17.303164 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:17.305138 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:17.305825 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:17.305969 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:17.307255 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:17.307263 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:17.307277 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:17.307804 (+ 527us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:17.307883 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:17.307907 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:17.308033 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:17.308409 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 01:22:17.308623 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":394,"threads_started":1}
I20260925 01:22:17.311537 10309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:17.312877 10309 log.cc:824] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:17.313745 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:17.316051 10309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d: No bootstrap required, opened a new log
I20260925 01:22:17.325289 10309 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd16e4c9a2541a9b7868b5d4752bd8d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39325 } }
I20260925 01:22:17.325757 10309 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:22:17.325810 10312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:17.318618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48929 (local address 127.20.102.190:39325)
0925 01:22:17.318779 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:17.318783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:17.319284 (+ 501us) server_negotiation.cc:407] Connection header received
0925 01:22:17.319484 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:17.319489 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:17.319561 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:17.319678 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:17.320193 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:17.321079 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:17.323165 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:17.323678 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:17.323845 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:17.325028 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:17.325033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:17.325036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:17.325052 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:17.325111 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:17.325127 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:17.325250 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:17.325445 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 01:22:17.325497 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 01:22:17.325908 10309 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbd16e4c9a2541a9b7868b5d4752bd8d, State: Initialized, Role: FOLLOWER
I20260925 01:22:17.326581 10309 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd16e4c9a2541a9b7868b5d4752bd8d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39325 } }
I20260925 01:22:17.326884 10309 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:22:17.327031 10309 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:22:17.327251 10309 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:17.329630 10309 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd16e4c9a2541a9b7868b5d4752bd8d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39325 } }
I20260925 01:22:17.330219 10309 leader_election.cc:302] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbd16e4c9a2541a9b7868b5d4752bd8d; no voters:
I20260925 01:22:17.330935 10309 leader_election.cc:288] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:22:17.331065 10319 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:17.331801 10319 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [term 1 LEADER]: Becoming Leader. State: Replica: bbd16e4c9a2541a9b7868b5d4752bd8d, State: Running, Role: LEADER
I20260925 01:22:17.332356 10319 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd16e4c9a2541a9b7868b5d4752bd8d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39325 } }
I20260925 01:22:17.333156 10309 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:17.335234 10321 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbd16e4c9a2541a9b7868b5d4752bd8d. Latest consensus state: current_term: 1 leader_uuid: "bbd16e4c9a2541a9b7868b5d4752bd8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd16e4c9a2541a9b7868b5d4752bd8d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39325 } } }
I20260925 01:22:17.335652 10321 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:17.335963 10320 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbd16e4c9a2541a9b7868b5d4752bd8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbd16e4c9a2541a9b7868b5d4752bd8d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39325 } } }
I20260925 01:22:17.336267 10320 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:17.341768 10328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:22:17.346839 10328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:17.357194 10328 catalog_manager.cc:1347] Generated new cluster ID: 5aa414578c944ea78b9c85f54bb200b8
I20260925 01:22:17.357339 10328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:17.366556 10328 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:17.367801 10328 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:17.374179 10328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbd16e4c9a2541a9b7868b5d4752bd8d: Generated new TSK 0
I20260925 01:22:17.375191 10328 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:22:17.381407 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10242
W20260925 01:22:17.383085 10304 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53965
W20260925 01:22:17.383912 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10242: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:22:17.384071 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 10242
W20260925 01:22:17.401762 10227 connection.cc:535] client connection to 127.20.102.190:39325 recv error: Network error: failed to read from TLS socket (remote: 127.20.102.190:39325): Connection reset by peer (error 104)
2026-09-25T01:22:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1920 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:17 dist-test-slave-s12s krb5kdc[10340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:17 dist-test-slave-s12s krb5kdc[10340](info): set up 2 sockets
Sep 25 01:22:17 dist-test-slave-s12s krb5kdc[10340](info): commencing operation
krb5kdc: starting...
W20260925 01:22:18.529410 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:18 dist-test-slave-s12s krb5kdc[10340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:22:18.780139 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39563
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:39021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:39563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:19.118865 10353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:19.119144 10353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:19.119354 10353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:19.126425 10353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:19.126551 10353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:19.126663 10353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:19.126777 10353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:19.126852 10353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:19.136272 10353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:39563
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:19.137599 10353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:19.138942 10353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:19.150887 10358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:19.150988 10359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:19.150895 10361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:19.152117 10353 server_base.cc:1034] running on GCE node
I20260925 01:22:19.152938 10353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:19.154214 10353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:19.155417 10353 hybrid_clock.cc:648] HybridClock initialized: now 1790299339155375 us; error 28 us; skew 500 ppm
I20260925 01:22:19.155766 10353 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:19 dist-test-slave-s12s krb5kdc[10340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299339, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:19.158977 10353 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:22:19.160455 10353 webserver.cc:460] Webserver started at http://127.20.102.190:32885/ using document root <none> and password file <none>
I20260925 01:22:19.161185 10353 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:19.161316 10353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:19.161634 10353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:19.164500 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "fa5effe8025a4c2dafd207fa7467c37d"
format_stamp: "Formatted at 2026-09-25 01:22:19 on dist-test-slave-s12s"
server_key: "68c41c9a6a6990e393144fb4d068fb45"
server_key_iv: "a6cc907d1c386f84155f0043089dbd87"
server_key_version: "encryptionkey@0"
I20260925 01:22:19.165313 10353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "fa5effe8025a4c2dafd207fa7467c37d"
format_stamp: "Formatted at 2026-09-25 01:22:19 on dist-test-slave-s12s"
server_key: "68c41c9a6a6990e393144fb4d068fb45"
server_key_iv: "a6cc907d1c386f84155f0043089dbd87"
server_key_version: "encryptionkey@0"
I20260925 01:22:19.170228 10353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 01:22:19.173519 10368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:19.174803 10353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:22:19.174983 10353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "fa5effe8025a4c2dafd207fa7467c37d"
format_stamp: "Formatted at 2026-09-25 01:22:19 on dist-test-slave-s12s"
server_key: "68c41c9a6a6990e393144fb4d068fb45"
server_key_iv: "a6cc907d1c386f84155f0043089dbd87"
server_key_version: "encryptionkey@0"
I20260925 01:22:19.175184 10353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:19.180845 10353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:19.185050 10353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:19.185344 10353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:19.203889 10353 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:39563
I20260925 01:22:19.203976 10419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:39563 every 8 connection(s)
I20260925 01:22:19.205601 10353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:19.209316 10420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:19.214372 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 10353
I20260925 01:22:19.214545 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:19.214996 20890 external_mini_cluster.cc:1428] Setting key 42ee36b04043bac9b93e659efa42d16f
I20260925 01:22:19.221679 10420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d: Bootstrap starting.
I20260925 01:22:19.224154 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:19.216862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36725 (local address 127.20.102.190:39563)
0925 01:22:19.217714 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:19.217735 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:19.217789 (+ 54us) server_negotiation.cc:407] Connection header received
0925 01:22:19.220289 (+ 2500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:19.220317 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:19.220761 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:19.221199 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:19.222179 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:19.222188 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:19.222206 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:19.222229 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:19.222301 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:19.222328 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:19.222540 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:19.222700 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 01:22:19.222878 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":582,"thread_start_us":361,"threads_started":1}
I20260925 01:22:19.225723 10420 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:19.227073 10420 log.cc:824] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:19.228030 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:19.230401 10420 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d: No bootstrap required, opened a new log
I20260925 01:22:19.235828 10423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:19.233179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50299 (local address 127.20.102.190:39563)
0925 01:22:19.233405 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:19.233414 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:19.233869 (+ 455us) server_negotiation.cc:407] Connection header received
0925 01:22:19.234037 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:19.234041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:19.234089 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:19.234184 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:19.235144 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:19.235149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:19.235152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:19.235166 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:19.235207 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:19.235224 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:19.235324 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:19.235467 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 01:22:19.235510 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:22:19.240299 10420 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5effe8025a4c2dafd207fa7467c37d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39563 } }
I20260925 01:22:19.240787 10420 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:22:19.240891 10420 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5effe8025a4c2dafd207fa7467c37d, State: Initialized, Role: FOLLOWER
I20260925 01:22:19.241472 10420 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5effe8025a4c2dafd207fa7467c37d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39563 } }
I20260925 01:22:19.241737 10420 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:22:19.241930 10420 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:22:19.242111 10420 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:19.244781 10420 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5effe8025a4c2dafd207fa7467c37d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39563 } }
I20260925 01:22:19.245285 10420 leader_election.cc:302] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa5effe8025a4c2dafd207fa7467c37d; no voters:
I20260925 01:22:19.245865 10420 leader_election.cc:288] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:22:19.246017 10430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:19.246694 10430 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [term 1 LEADER]: Becoming Leader. State: Replica: fa5effe8025a4c2dafd207fa7467c37d, State: Running, Role: LEADER
I20260925 01:22:19.247197 10430 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5effe8025a4c2dafd207fa7467c37d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39563 } }
I20260925 01:22:19.248013 10420 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:19.250474 10432 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa5effe8025a4c2dafd207fa7467c37d. Latest consensus state: current_term: 1 leader_uuid: "fa5effe8025a4c2dafd207fa7467c37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5effe8025a4c2dafd207fa7467c37d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39563 } } }
I20260925 01:22:19.251034 10432 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:19.250922 10431 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa5effe8025a4c2dafd207fa7467c37d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5effe8025a4c2dafd207fa7467c37d" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39563 } } }
I20260925 01:22:19.252346 10431 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:19.253913 10439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:22:19.259289 10439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:19.268486 10439 catalog_manager.cc:1347] Generated new cluster ID: d8b47e7af0df4e52b56fb6784274c271
I20260925 01:22:19.268643 10439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:19.280547 10439 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:19.280998 10439 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:19.288620 10439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa5effe8025a4c2dafd207fa7467c37d: Generated new TSK 0
I20260925 01:22:19.289474 10439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:22:19.297588 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10353
W20260925 01:22:19.299124 10415 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36725
W20260925 01:22:19.299871 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10353: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:22:19.300037 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 10353
2026-09-25T01:22:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1917 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:19 dist-test-slave-s12s krb5kdc[10451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:19 dist-test-slave-s12s krb5kdc[10451](info): set up 2 sockets
Sep 25 01:22:19 dist-test-slave-s12s krb5kdc[10451](info): commencing operation
krb5kdc: starting...
W20260925 01:22:20.425698 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:20 dist-test-slave-s12s krb5kdc[10451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:22:20.681706 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:40511
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:35133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:40511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:21.021534 10464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:21.021826 10464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:21.022033 10464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:21.029496 10464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:21.029609 10464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:21.029686 10464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:21.029801 10464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:21.029886 10464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:21.040354 10464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:35133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:40511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:40511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:21.041705 10464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:21.043072 10464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:21.055012 10472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:21.055117 10469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:21.055090 10470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:21.056159 10464 server_base.cc:1034] running on GCE node
I20260925 01:22:21.057010 10464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:21.058458 10464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:21.059697 10464 hybrid_clock.cc:648] HybridClock initialized: now 1790299341059650 us; error 44 us; skew 500 ppm
I20260925 01:22:21.060061 10464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:21 dist-test-slave-s12s krb5kdc[10451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299341, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:21.063284 10464 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:22:21.065021 10464 webserver.cc:460] Webserver started at http://127.20.102.190:44107/ using document root <none> and password file <none>
I20260925 01:22:21.065693 10464 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:21.065794 10464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:21.066036 10464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:21.069058 10464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "ab98bc3da9f14db396559a5e636400f3"
format_stamp: "Formatted at 2026-09-25 01:22:21 on dist-test-slave-s12s"
server_key: "3f817e1a07c03063e397d6cce3123f73"
server_key_iv: "dab64a2fec846897c413300d70ee59e6"
server_key_version: "encryptionkey@0"
I20260925 01:22:21.069764 10464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "ab98bc3da9f14db396559a5e636400f3"
format_stamp: "Formatted at 2026-09-25 01:22:21 on dist-test-slave-s12s"
server_key: "3f817e1a07c03063e397d6cce3123f73"
server_key_iv: "dab64a2fec846897c413300d70ee59e6"
server_key_version: "encryptionkey@0"
I20260925 01:22:21.074836 10464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 01:22:21.078234 10479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:21.079540 10464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:22:21.079720 10464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "ab98bc3da9f14db396559a5e636400f3"
format_stamp: "Formatted at 2026-09-25 01:22:21 on dist-test-slave-s12s"
server_key: "3f817e1a07c03063e397d6cce3123f73"
server_key_iv: "dab64a2fec846897c413300d70ee59e6"
server_key_version: "encryptionkey@0"
I20260925 01:22:21.079912 10464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:21.100239 10464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:21.104259 10464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:21.104550 10464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:21.123955 10464 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:40511
I20260925 01:22:21.123982 10530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:40511 every 8 connection(s)
I20260925 01:22:21.125481 10464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:21.126333 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 10464
I20260925 01:22:21.126530 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:21.127154 20890 external_mini_cluster.cc:1428] Setting key 15ab54302dea1a49c9bdfce6c9381559
I20260925 01:22:21.130777 10531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 01:22:21 dist-test-slave-s12s krb5kdc[10451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:22:21.143427 10531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3: Bootstrap starting.
I20260925 01:22:21.147678 10531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:21.149021 10531 log.cc:824] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:21.152070 10531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3: No bootstrap required, opened a new log
I20260925 01:22:21.152565 10534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:21.129201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39837 (local address 127.20.102.190:40511)
0925 01:22:21.130003 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:21.130019 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:21.130079 (+ 60us) server_negotiation.cc:407] Connection header received
0925 01:22:21.132474 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:21.132502 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:21.133077 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:21.133507 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:21.134851 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:21.137020 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:21.138341 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:21.139356 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:21.139564 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:21.144072 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:21.144083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:21.144102 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:21.144144 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:21.148266 (+ 4122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:21.149066 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:21.149075 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:21.149083 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:21.149293 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:21.149726 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:21.149730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:21.149732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:21.150293 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:21.150625 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:21.151073 (+ 448us) server_negotiation.cc:299] Negotiation successful
0925 01:22:21.151508 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":327,"threads_started":1}
I20260925 01:22:21.156599 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:21.160751 10531 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab98bc3da9f14db396559a5e636400f3" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 40511 } }
I20260925 01:22:21.161288 10531 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:22:21.161427 10531 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab98bc3da9f14db396559a5e636400f3, State: Initialized, Role: FOLLOWER
I20260925 01:22:21.162081 10531 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab98bc3da9f14db396559a5e636400f3" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 40511 } }
I20260925 01:22:21.162328 10531 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:22:21.162480 10531 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:22:21.162665 10531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:21.164999 10531 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab98bc3da9f14db396559a5e636400f3" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 40511 } }
I20260925 01:22:21.165511 10531 leader_election.cc:302] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab98bc3da9f14db396559a5e636400f3; no voters:
I20260925 01:22:21.166270 10531 leader_election.cc:288] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:22:21.166384 10541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:21.168790 10541 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [term 1 LEADER]: Becoming Leader. State: Replica: ab98bc3da9f14db396559a5e636400f3, State: Running, Role: LEADER
I20260925 01:22:21.169291 10541 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab98bc3da9f14db396559a5e636400f3" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 40511 } }
I20260925 01:22:21.169567 10531 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:21.174593 10542 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab98bc3da9f14db396559a5e636400f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab98bc3da9f14db396559a5e636400f3" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 40511 } } }
I20260925 01:22:21.174575 10543 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab98bc3da9f14db396559a5e636400f3. Latest consensus state: current_term: 1 leader_uuid: "ab98bc3da9f14db396559a5e636400f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab98bc3da9f14db396559a5e636400f3" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 40511 } } }
I20260925 01:22:21.175027 10542 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:21.175060 10543 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3 [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:21.179662 10534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:21.160907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35477 (local address 127.20.102.190:40511)
0925 01:22:21.161115 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:21.161123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:21.161589 (+ 466us) server_negotiation.cc:407] Connection header received
0925 01:22:21.161801 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:21.161816 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:21.161889 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:21.161990 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:21.163675 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:21.164406 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:21.165862 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:21.166414 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:21.166576 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:21.169925 (+ 3349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:21.169930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:21.169935 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:21.169971 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:21.174614 (+ 4643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:21.175752 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:21.175761 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:21.175767 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:21.175997 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:21.176595 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:21.176602 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:21.176608 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:21.177113 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:21.178707 (+ 1594us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:21.178892 (+ 185us) server_negotiation.cc:299] Negotiation successful
0925 01:22:21.179132 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 01:22:21.182544 10548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:22:21.187906 10548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:21.196738 10548 catalog_manager.cc:1347] Generated new cluster ID: 309e9bdbc146439b8f41d4c7ffffe500
I20260925 01:22:21.196861 10548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:21.210285 10548 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:21.211690 10548 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:21.222894 10548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab98bc3da9f14db396559a5e636400f3: Generated new TSK 0
I20260925 01:22:21.223800 10548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:22:21.229525 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10464
I20260925 01:22:21.231453 10526 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:22:21.322443 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 10464
2026-09-25T01:22:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2018 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:21 dist-test-slave-s12s krb5kdc[10563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:21 dist-test-slave-s12s krb5kdc[10563](info): set up 2 sockets
Sep 25 01:22:21 dist-test-slave-s12s krb5kdc[10563](info): commencing operation
krb5kdc: starting...
W20260925 01:22:22.421331 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:22 dist-test-slave-s12s krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:22:22.687963 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39401
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:42201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:39401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:23.024322 10576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:23.024637 10576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:23.024871 10576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:23.032177 10576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:23.032317 10576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:23.032433 10576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:23.032542 10576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:23.032634 10576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:23.042932 10576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:42201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:39401
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:39401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:23.044306 10576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:23.045660 10576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:23.057879 10581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:23.057900 10582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:23.057848 10584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:23.058780 10576 server_base.cc:1034] running on GCE node
I20260925 01:22:23.059582 10576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:23.061111 10576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:23.062340 10576 hybrid_clock.cc:648] HybridClock initialized: now 1790299343062289 us; error 36 us; skew 500 ppm
I20260925 01:22:23.062757 10576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:23 dist-test-slave-s12s krb5kdc[10563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299343, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:23.071806 10576 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:22:23.073557 10576 webserver.cc:460] Webserver started at http://127.20.102.190:33521/ using document root <none> and password file <none>
I20260925 01:22:23.074249 10576 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:23.074355 10576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:23.074626 10576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:23.077817 10576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "23d760d38450432f9184b398758ad59f"
format_stamp: "Formatted at 2026-09-25 01:22:23 on dist-test-slave-s12s"
server_key: "27ded4b838dd959049cdaab045e4e596"
server_key_iv: "99809e7e2c6c6597ba974d02bc8f96fc"
server_key_version: "encryptionkey@0"
I20260925 01:22:23.078544 10576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "23d760d38450432f9184b398758ad59f"
format_stamp: "Formatted at 2026-09-25 01:22:23 on dist-test-slave-s12s"
server_key: "27ded4b838dd959049cdaab045e4e596"
server_key_iv: "99809e7e2c6c6597ba974d02bc8f96fc"
server_key_version: "encryptionkey@0"
I20260925 01:22:23.083933 10576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 01:22:23.087546 10591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:23.089025 10576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 01:22:23.089175 10576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "23d760d38450432f9184b398758ad59f"
format_stamp: "Formatted at 2026-09-25 01:22:23 on dist-test-slave-s12s"
server_key: "27ded4b838dd959049cdaab045e4e596"
server_key_iv: "99809e7e2c6c6597ba974d02bc8f96fc"
server_key_version: "encryptionkey@0"
I20260925 01:22:23.089349 10576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:23.112973 10576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:23.117576 10576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:23.117825 10576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:23.136860 10576 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:39401
I20260925 01:22:23.136905 10642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:39401 every 8 connection(s)
I20260925 01:22:23.138278 10576 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:23.142195 10643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:23.143116 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 10576
I20260925 01:22:23.143308 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:23.143777 20890 external_mini_cluster.cc:1428] Setting key 0df4fe9212f7bfba63e7809a6fcecfbc
I20260925 01:22:23.156185 10643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f: Bootstrap starting.
I20260925 01:22:23.157441 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:23.145594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43091 (local address 127.20.102.190:39401)
0925 01:22:23.146367 (+ 773us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:23.146378 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:23.146426 (+ 48us) server_negotiation.cc:407] Connection header received
0925 01:22:23.148280 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:23.148327 (+ 47us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:23.148795 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:23.149142 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:23.149788 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:23.151012 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:23.152958 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:23.153690 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:23.153822 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:23.155131 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:23.155139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:23.155153 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:23.155734 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:23.155816 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:23.155841 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:23.155926 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:23.156170 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 01:22:23.156443 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":347,"threads_started":1}
I20260925 01:22:23.159523 10643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:23.160765 10643 log.cc:824] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:23.161193 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:23.163527 10643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f: No bootstrap required, opened a new log
I20260925 01:22:23.172278 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:23.165795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57769 (local address 127.20.102.190:39401)
0925 01:22:23.166019 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:23.166025 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:23.166407 (+ 382us) server_negotiation.cc:407] Connection header received
0925 01:22:23.166538 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:23.166543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:23.166624 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:23.166731 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:23.167210 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:23.168286 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:23.169663 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:23.170379 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:23.170509 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:23.171517 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:23.171523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:23.171530 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:23.171551 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:23.171613 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:23.171637 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:23.171736 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:23.171852 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 01:22:23.171944 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:22:23.173170 10643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d760d38450432f9184b398758ad59f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39401 } }
I20260925 01:22:23.173596 10643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:22:23.173717 10643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d760d38450432f9184b398758ad59f, State: Initialized, Role: FOLLOWER
I20260925 01:22:23.174283 10643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d760d38450432f9184b398758ad59f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39401 } }
I20260925 01:22:23.174494 10643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:22:23.174597 10643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:22:23.174727 10643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:23.176946 10643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d760d38450432f9184b398758ad59f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39401 } }
I20260925 01:22:23.177410 10643 leader_election.cc:302] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23d760d38450432f9184b398758ad59f; no voters:
I20260925 01:22:23.177943 10643 leader_election.cc:288] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:22:23.178066 10653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:23.178742 10653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [term 1 LEADER]: Becoming Leader. State: Replica: 23d760d38450432f9184b398758ad59f, State: Running, Role: LEADER
I20260925 01:22:23.179385 10653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d760d38450432f9184b398758ad59f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39401 } }
I20260925 01:22:23.180192 10643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:23.186244 10654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23d760d38450432f9184b398758ad59f. Latest consensus state: current_term: 1 leader_uuid: "23d760d38450432f9184b398758ad59f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d760d38450432f9184b398758ad59f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39401 } } }
I20260925 01:22:23.186409 10655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23d760d38450432f9184b398758ad59f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23d760d38450432f9184b398758ad59f" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 39401 } } }
I20260925 01:22:23.186944 10654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:23.186941 10655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:23.189498 10668 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 01:22:23.193840 10664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:22:23.198076 10664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:23.206640 10664 catalog_manager.cc:1347] Generated new cluster ID: 8e05374dd3bd4278b705ef1383ad2e6d
I20260925 01:22:23.206755 10664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:23.231364 10664 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:23.233541 10664 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:23.241133 10664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23d760d38450432f9184b398758ad59f: Generated new TSK 0
I20260925 01:22:23.242201 10664 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:22:23.251125 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10576
W20260925 01:22:23.252211 10638 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43091
W20260925 01:22:23.252969 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10576: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:22:23.253130 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 10576
2026-09-25T01:22:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1932 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:23 dist-test-slave-s12s krb5kdc[10674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:23 dist-test-slave-s12s krb5kdc[10674](info): set up 2 sockets
Sep 25 01:22:23 dist-test-slave-s12s krb5kdc[10674](info): commencing operation
krb5kdc: starting...
W20260925 01:22:24.449564 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.160s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:24 dist-test-slave-s12s krb5kdc[10674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:22:24.711797 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:44065
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:39761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:44065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:25.048105 10687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:25.048463 10687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:25.048645 10687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:25.055843 10687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:25.055997 10687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:25.056118 10687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:25.056231 10687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:25.056314 10687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:25.066437 10687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:44065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:44065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:25.067780 10687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:25.069136 10687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:25.081264 10692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:25.081197 10695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:25.081189 10693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:25.082019 10687 server_base.cc:1034] running on GCE node
I20260925 01:22:25.082794 10687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:25.084149 10687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:25.085386 10687 hybrid_clock.cc:648] HybridClock initialized: now 1790299345085302 us; error 68 us; skew 500 ppm
I20260925 01:22:25.085765 10687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:25 dist-test-slave-s12s krb5kdc[10674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299345, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:25.089048 10687 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:22:25.090607 10687 webserver.cc:460] Webserver started at http://127.20.102.190:34753/ using document root <none> and password file <none>
I20260925 01:22:25.091313 10687 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:25.091442 10687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:25.091753 10687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:25.094801 10687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "9196bd580c1242a4a41041d696e2ab74"
format_stamp: "Formatted at 2026-09-25 01:22:25 on dist-test-slave-s12s"
server_key: "b740ce347d08ab2d01c22b70f05864aa"
server_key_iv: "1a5fee47188f5d5f323e130f1a2e3c52"
server_key_version: "encryptionkey@0"
I20260925 01:22:25.095715 10687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "9196bd580c1242a4a41041d696e2ab74"
format_stamp: "Formatted at 2026-09-25 01:22:25 on dist-test-slave-s12s"
server_key: "b740ce347d08ab2d01c22b70f05864aa"
server_key_iv: "1a5fee47188f5d5f323e130f1a2e3c52"
server_key_version: "encryptionkey@0"
I20260925 01:22:25.101009 10687 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:22:25.104525 10702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:25.106211 10687 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:22:25.106433 10687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "9196bd580c1242a4a41041d696e2ab74"
format_stamp: "Formatted at 2026-09-25 01:22:25 on dist-test-slave-s12s"
server_key: "b740ce347d08ab2d01c22b70f05864aa"
server_key_iv: "1a5fee47188f5d5f323e130f1a2e3c52"
server_key_version: "encryptionkey@0"
I20260925 01:22:25.106678 10687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:25.113268 10687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:25.117962 10687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:25.118296 10687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:25.138880 10687 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:44065
I20260925 01:22:25.138978 10753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:44065 every 8 connection(s)
I20260925 01:22:25.140623 10687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:25.144410 10754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:25.146775 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 10687
I20260925 01:22:25.146976 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:25.147470 20890 external_mini_cluster.cc:1428] Setting key 9d6ae41e572281072be8015ada724e80
I20260925 01:22:25.155037 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:25.149251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33601 (local address 127.20.102.190:44065)
0925 01:22:25.150070 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:25.150081 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:25.150122 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:22:25.151785 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:25.151807 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:25.152141 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:25.152468 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:25.153519 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:25.153525 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:25.153537 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:25.153550 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:25.153618 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:25.153640 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:25.153810 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:25.153912 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 01:22:25.154146 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":367,"threads_started":1}
I20260925 01:22:25.156459 10754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74: Bootstrap starting.
I20260925 01:22:25.158834 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:25.159866 10754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:25.160909 10754 log.cc:824] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:25.164012 10754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74: No bootstrap required, opened a new log
I20260925 01:22:25.165692 10757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:25.163106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32789 (local address 127.20.102.190:44065)
0925 01:22:25.163274 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:25.163278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:25.163769 (+ 491us) server_negotiation.cc:407] Connection header received
0925 01:22:25.163966 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:25.163971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:25.164019 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:25.164127 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:25.165045 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:25.165051 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:25.165064 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 01:22:25.165092 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:25.165130 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 01:22:25.165144 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:25.165262 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:25.165355 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 01:22:25.165421 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 01:22:25.172050 10754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9196bd580c1242a4a41041d696e2ab74" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44065 } }
I20260925 01:22:25.172430 10754 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 01:22:25.172545 10754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9196bd580c1242a4a41041d696e2ab74, State: Initialized, Role: FOLLOWER
I20260925 01:22:25.173110 10754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9196bd580c1242a4a41041d696e2ab74" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44065 } }
I20260925 01:22:25.173352 10754 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 01:22:25.173519 10754 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 01:22:25.173691 10754 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:25.176205 10754 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9196bd580c1242a4a41041d696e2ab74" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44065 } }
I20260925 01:22:25.176745 10754 leader_election.cc:302] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9196bd580c1242a4a41041d696e2ab74; no voters:
I20260925 01:22:25.177353 10754 leader_election.cc:288] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 01:22:25.177527 10764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:25.178254 10764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [term 1 LEADER]: Becoming Leader. State: Replica: 9196bd580c1242a4a41041d696e2ab74, State: Running, Role: LEADER
I20260925 01:22:25.178846 10764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9196bd580c1242a4a41041d696e2ab74" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44065 } }
I20260925 01:22:25.179417 10754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:25.181731 10765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9196bd580c1242a4a41041d696e2ab74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9196bd580c1242a4a41041d696e2ab74" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44065 } } }
I20260925 01:22:25.182175 10765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:25.182740 10766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9196bd580c1242a4a41041d696e2ab74. Latest consensus state: current_term: 1 leader_uuid: "9196bd580c1242a4a41041d696e2ab74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9196bd580c1242a4a41041d696e2ab74" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44065 } } }
I20260925 01:22:25.183051 10766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74 [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:25.184401 10771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 01:22:25.189644 10771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:25.199388 10771 catalog_manager.cc:1347] Generated new cluster ID: 1bb57cadcf82401fbb3cc0daeefc3809
I20260925 01:22:25.199553 10771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:25.213501 10771 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:25.213930 10771 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:25.225183 10771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9196bd580c1242a4a41041d696e2ab74: Generated new TSK 0
I20260925 01:22:25.226033 10771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 01:22:25.241977 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10687
W20260925 01:22:25.243701 10749 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33601
W20260925 01:22:25.244427 20890 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10687: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 01:22:25.244580 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 10687
2026-09-25T01:22:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1991 ms)
[----------] 18 tests from AuthTokenIssuingTest (35410 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:25 dist-test-slave-s12s krb5kdc[10785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:25 dist-test-slave-s12s krb5kdc[10785](info): set up 2 sockets
Sep 25 01:22:25 dist-test-slave-s12s krb5kdc[10785](info): commencing operation
krb5kdc: starting...
W20260925 01:22:26.429458 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:26 dist-test-slave-s12s krb5kdc[10785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:22:26.700073 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:44697
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:44697,127.20.102.189:39623,127.20.102.188:43051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:27.042228 10798 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:22:27.042595 10798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:27.042747 10798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:27.042883 10798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:27.050203 10798 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:27.050330 10798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:27.050439 10798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:27.050530 10798 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:27.050622 10798 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:27.060412 10798 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:44697,127.20.102.189:39623,127.20.102.188:43051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:44697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:27.061777 10798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:27.063119 10798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:27.075691 10803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:27.075784 10804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:27.075752 10806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:27.076571 10798 server_base.cc:1034] running on GCE node
I20260925 01:22:27.077347 10798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:27.078719 10798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:27.079970 10798 hybrid_clock.cc:648] HybridClock initialized: now 1790299347079911 us; error 33 us; skew 500 ppm
I20260925 01:22:27.080400 10798 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:27 dist-test-slave-s12s krb5kdc[10785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:27.083659 10798 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:22:27.085071 10798 webserver.cc:460] Webserver started at http://127.20.102.190:38899/ using document root <none> and password file <none>
I20260925 01:22:27.085726 10798 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:27.085810 10798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:27.086071 10798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:27.089064 10798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "fd70540803a747308a92752104bc9dbd"
format_stamp: "Formatted at 2026-09-25 01:22:27 on dist-test-slave-s12s"
server_key: "59a90a6bcf1ee026e712e080aeb17352"
server_key_iv: "a4ac0de116b208e090fb1e45829e42da"
server_key_version: "encryptionkey@0"
I20260925 01:22:27.089797 10798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "fd70540803a747308a92752104bc9dbd"
format_stamp: "Formatted at 2026-09-25 01:22:27 on dist-test-slave-s12s"
server_key: "59a90a6bcf1ee026e712e080aeb17352"
server_key_iv: "a4ac0de116b208e090fb1e45829e42da"
server_key_version: "encryptionkey@0"
I20260925 01:22:27.094954 10798 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:22:27.098536 10813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:27.099999 10798 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 01:22:27.100174 10798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "fd70540803a747308a92752104bc9dbd"
format_stamp: "Formatted at 2026-09-25 01:22:27 on dist-test-slave-s12s"
server_key: "59a90a6bcf1ee026e712e080aeb17352"
server_key_iv: "a4ac0de116b208e090fb1e45829e42da"
server_key_version: "encryptionkey@0"
I20260925 01:22:27.100389 10798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:27.137126 10798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:27.141355 10798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:27.141604 10798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:27.160208 10798 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:44697
I20260925 01:22:27.160243 10864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:44697 every 8 connection(s)
I20260925 01:22:27.161557 10798 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:27.165452 10865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:27.166760 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 10798
I20260925 01:22:27.166978 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:27.167817 20890 external_mini_cluster.cc:1428] Setting key 73832041e534ca0ccd38caaa849b5978
I20260925 01:22:27.175031 10865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:22:27 dist-test-slave-s12s krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.102.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.189@KRBTEST.COM" created.
I20260925 01:22:27.203212 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.179260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:44697 (local address 127.0.0.1:40994)
0925 01:22:27.180241 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.180298 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:27.183300 (+ 3002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:27.184203 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:27.184212 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:27.185065 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:27.185574 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:27.185601 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.187460 (+ 1859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.187469 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:27.189339 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:27.189355 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.190431 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.190441 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:27.190655 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:27.191398 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:27.191432 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:27.194159 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:27.198560 (+ 4401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:27.198570 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:27.198580 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:27.198976 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:27.199501 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:27.199504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:27.199506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:27.199653 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:27.200725 (+ 1072us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:27.200735 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:27.200966 (+ 231us) client_negotiation.cc:769] Sending connection context
0925 01:22:27.201215 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 01:22:27.201837 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":178,"thread_start_us":431,"threads_started":1}
I20260925 01:22:27.203235 10869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.179260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40994 (local address 127.20.102.190:44697)
0925 01:22:27.181618 (+ 2358us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:27.181625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:27.181676 (+ 51us) server_negotiation.cc:407] Connection header received
0925 01:22:27.183701 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:27.183708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:27.183945 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:27.184118 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:27.185825 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.187279 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.189522 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.190177 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.190341 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:27.194440 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:27.194454 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:27.194475 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:27.194549 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:27.198424 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:27.199232 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:27.199239 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:27.199241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:27.199398 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:27.199846 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:27.199853 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:27.199856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:27.200549 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:27.200797 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:27.201339 (+ 542us) server_negotiation.cc:299] Negotiation successful
0925 01:22:27.201729 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":2110,"spinlock_wait_cycles":25472,"thread_start_us":1849,"threads_started":1}
I20260925 01:22:27.206331 10865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:22:27.207508 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.207099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:39623 (local address 127.0.0.1:59268)
0925 01:22:27.207282 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.207401 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 01:22:27.207832 10817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
I20260925 01:22:27.209563 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.209230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:39623 (local address 127.0.0.1:59276)
0925 01:22:27.209378 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.209440 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 01:22:27.209821 10865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:39623: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.189.keytab.
Entry for principal kudu/127.20.102.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.189.keytab.
I20260925 01:22:27.230049 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.189:39623
--webserver_interface=127.20.102.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.189.keytab
--principal=kudu/127.20.102.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:44697,127.20.102.189:39623,127.20.102.188:43051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:22:27.259193 10865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } attempt: 1
I20260925 01:22:27.260402 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.259919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:39623 (local address 127.0.0.1:59282)
0925 01:22:27.260134 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.260263 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 01:22:27.261472 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.261132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:39623 (local address 127.0.0.1:59298)
0925 01:22:27.261281 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.261349 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 01:22:27.261740 10865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:39623: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
I20260925 01:22:27.330013 10865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } attempt: 2
I20260925 01:22:27.331202 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.330709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:39623 (local address 127.0.0.1:59308)
0925 01:22:27.330968 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.331063 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 01:22:27.332314 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.331942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:39623 (local address 127.0.0.1:59310)
0925 01:22:27.332118 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.332186 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 01:22:27.332583 10865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:39623: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
I20260925 01:22:27.423857 10865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } attempt: 3
I20260925 01:22:27.425130 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.424656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:39623 (local address 127.0.0.1:59322)
0925 01:22:27.424893 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.424987 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260925 01:22:27.426153 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.425829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:39623 (local address 127.0.0.1:59336)
0925 01:22:27.425961 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.426028 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 01:22:27.426384 10865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:39623: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
W20260925 01:22:27.565779 10872 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:22:27.566197 10872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:27.566329 10872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:27.566454 10872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 01:22:27.569635 10865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } attempt: 4
I20260925 01:22:27.570894 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.570327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:39623 (local address 127.0.0.1:59342)
0925 01:22:27.570670 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.570758 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260925 01:22:27.571971 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.571623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:39623 (local address 127.0.0.1:59352)
0925 01:22:27.571756 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.571837 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 01:22:27.572248 10865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:39623: Network error: Client connection negotiation failed: client connection to 127.20.102.189:39623: connect: Connection refused (error 111)
W20260925 01:22:27.574788 10872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:27.574869 10872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:27.575009 10872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:27.575089 10872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:27.575196 10872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:27.585642 10872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:44697,127.20.102.189:39623,127.20.102.188:43051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.189.keytab
--principal=kudu/127.20.102.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.189:39623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.102.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:27.586983 10872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:27.588280 10872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:27.600365 10880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:27.600332 10878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:27.600293 10877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:27.601469 10872 server_base.cc:1034] running on GCE node
I20260925 01:22:27.602285 10872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:27.603739 10872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:27.604993 10872 hybrid_clock.cc:648] HybridClock initialized: now 1790299347604930 us; error 37 us; skew 500 ppm
I20260925 01:22:27.605401 10872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:27 dist-test-slave-s12s krb5kdc[10785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:27.608855 10872 init.cc:377] Logged in from keytab as kudu/127.20.102.189@KRBTEST.COM (short username kudu)
I20260925 01:22:27.610466 10872 webserver.cc:460] Webserver started at http://127.20.102.189:38635/ using document root <none> and password file <none>
I20260925 01:22:27.611217 10872 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:27.611349 10872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:27.611688 10872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:27.614778 10872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/data/instance:
uuid: "67359673c33b4aa681b62297349cd07a"
format_stamp: "Formatted at 2026-09-25 01:22:27 on dist-test-slave-s12s"
server_key: "293ca450bf6d4495d33f669705bee86e"
server_key_iv: "9b4d6af8647be08d3ec150ed60fc138b"
server_key_version: "encryptionkey@0"
I20260925 01:22:27.615622 10872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/wal/instance:
uuid: "67359673c33b4aa681b62297349cd07a"
format_stamp: "Formatted at 2026-09-25 01:22:27 on dist-test-slave-s12s"
server_key: "293ca450bf6d4495d33f669705bee86e"
server_key_iv: "9b4d6af8647be08d3ec150ed60fc138b"
server_key_version: "encryptionkey@0"
I20260925 01:22:27.621068 10872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 01:22:27.624531 10887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:27.625967 10872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:22:27.626169 10872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/wal
uuid: "67359673c33b4aa681b62297349cd07a"
format_stamp: "Formatted at 2026-09-25 01:22:27 on dist-test-slave-s12s"
server_key: "293ca450bf6d4495d33f669705bee86e"
server_key_iv: "9b4d6af8647be08d3ec150ed60fc138b"
server_key_version: "encryptionkey@0"
I20260925 01:22:27.626399 10872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:27.644838 10872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:27.649425 10872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:27.649722 10872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:27.668577 10872 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.189:39623
I20260925 01:22:27.668586 10938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.189:39623 every 8 connection(s)
I20260925 01:22:27.669961 10872 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/data/info.pb
I20260925 01:22:27.673736 10939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:27.675993 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 10872
I20260925 01:22:27.676190 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-1/wal/instance
I20260925 01:22:27.676748 20890 external_mini_cluster.cc:1428] Setting key 03168e7a95476ebff9154cbd2f94c244
I20260925 01:22:27.683207 10939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:22:27 dist-test-slave-s12s krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.189@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.102.188@KRBTEST.COM; defaulting to no policy
I20260925 01:22:27.708868 10869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.686558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41004 (local address 127.20.102.190:44697)
0925 01:22:27.686787 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:27.686795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:27.688100 (+ 1305us) server_negotiation.cc:407] Connection header received
0925 01:22:27.691120 (+ 3020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:27.691143 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:27.691215 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:27.691306 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:27.693464 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.694179 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.696640 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.697109 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.697219 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:27.701913 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:27.701919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:27.701922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:27.701960 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:27.705010 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:27.706039 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:27.706045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:27.706048 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:27.706175 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:27.706712 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:27.706719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:27.706724 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:27.707142 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:27.707315 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:27.708353 (+ 1038us) server_negotiation.cc:299] Negotiation successful
0925 01:22:27.708562 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.188@KRBTEST.COM" created.
I20260925 01:22:27.709794 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.686995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:44697 (local address 127.0.0.1:41004)
0925 01:22:27.687874 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.687926 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:27.690811 (+ 2885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:27.691575 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:27.691589 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:27.692408 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:27.693253 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:27.693280 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.694348 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.694357 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:27.696489 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:27.696507 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.697346 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.697355 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:27.697710 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:27.698529 (+ 819us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:27.698562 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:27.701642 (+ 3080us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:27.705163 (+ 3521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:27.705174 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:27.705194 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:27.705918 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:27.706294 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:27.706301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:27.706307 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:27.706559 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:27.707402 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:27.707412 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:27.707916 (+ 504us) client_negotiation.cc:769] Sending connection context
0925 01:22:27.708174 (+ 258us) client_negotiation.cc:240] Negotiation successful
0925 01:22:27.708600 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":343,"threads_started":1}
I20260925 01:22:27.711647 10939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:22:27 dist-test-slave-s12s krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.189@KRBTEST.COM for kudu/127.20.102.189@KRBTEST.COM
I20260925 01:22:27.728346 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.712464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:39623 (local address 127.0.0.1:59358)
0925 01:22:27.712647 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.712667 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:27.712800 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:27.713777 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:27.713781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:27.714215 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:27.714343 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:27.714351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.715819 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.715827 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:27.717368 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:27.717387 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.718557 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.718565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:27.718800 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:27.719400 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:27.719432 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:27.721605 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:27.725935 (+ 4330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:27.725939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:27.725942 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:27.726304 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:27.726733 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:27.726736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:27.726738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:27.726876 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:27.727643 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:27.727647 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:27.727761 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 01:22:27.727868 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 01:22:27.728084 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 01:22:27.728914 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.712741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59358 (local address 127.20.102.189:39623)
0925 01:22:27.713378 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:27.713383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:27.713406 (+ 23us) server_negotiation.cc:407] Connection header received
0925 01:22:27.713447 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:27.713452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:27.713626 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:27.713786 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:27.714508 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.715658 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.717510 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.718337 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.718478 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:27.721829 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:27.721845 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:27.721865 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:27.721913 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:27.725778 (+ 3865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:27.726454 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:27.726465 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:27.726471 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:27.726623 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:27.726993 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:27.727000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:27.727006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:27.727506 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:27.727725 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:27.728080 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 01:22:27.728389 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":364,"threads_started":1}
I20260925 01:22:27.731387 10939 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:22:27.732473 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.732049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39010)
0925 01:22:27.732210 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.732321 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 01:22:27.732846 10891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
I20260925 01:22:27.734444 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.734139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39016)
0925 01:22:27.734275 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.734350 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 01:22:27.734745 10939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:43051: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.188.keytab.
Entry for principal kudu/127.20.102.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.188.keytab.
I20260925 01:22:27.745427 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.188:43051
--webserver_interface=127.20.102.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.188.keytab
--principal=kudu/127.20.102.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:45401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:44697,127.20.102.189:39623,127.20.102.188:43051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:22:27.784101 10939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } attempt: 1
I20260925 01:22:27.785473 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.784964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39026)
0925 01:22:27.785182 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.785312 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 01:22:27.786689 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.786256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39040)
0925 01:22:27.786451 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.786574 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 01:22:27.786963 10939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:43051: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
I20260925 01:22:27.855194 10939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } attempt: 2
I20260925 01:22:27.856339 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.855880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39052)
0925 01:22:27.856085 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.856182 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 01:22:27.857393 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.857083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39062)
0925 01:22:27.857222 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.857291 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 01:22:27.857590 10939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:43051: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
I20260925 01:22:27.871529 10865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } attempt: 5
Sep 25 01:22:27 dist-test-slave-s12s krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for kudu/127.20.102.189@KRBTEST.COM
I20260925 01:22:27.887506 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.872252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:39623 (local address 127.0.0.1:59364)
0925 01:22:27.872471 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.872493 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:27.872628 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:27.873156 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:27.873162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:27.873657 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:27.873888 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:27.873901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.874994 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.875001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:27.876446 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:27.876463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.877375 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.877385 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:27.877671 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:27.878399 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:27.878422 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:27.880958 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:27.885038 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:27.885042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:27.885051 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:27.885447 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:27.885857 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:27.885860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:27.885862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:27.886011 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:27.886749 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:27.886752 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:27.886884 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 01:22:27.887033 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 01:22:27.887252 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 01:22:27.887732 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.872537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.20.102.189:39623)
0925 01:22:27.872802 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:27.872807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:27.872827 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:22:27.872887 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:27.872891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:27.872974 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:27.873076 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:27.874076 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.874830 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.876620 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:27.877193 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:27.877309 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:27.881190 (+ 3881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:27.881197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:27.881202 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:27.881244 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:27.884889 (+ 3645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:27.885585 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:27.885590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:27.885593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:27.885746 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:27.886122 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:27.886130 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:27.886135 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:27.886605 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:27.886762 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:27.887178 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 01:22:27.887420 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:22:27.888818 10865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:22:27.889847 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.889484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39076)
0925 01:22:27.889643 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.889718 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 01:22:27.890723 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.890403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39078)
0925 01:22:27.890567 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.890637 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 01:22:27.890923 10865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:43051: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
I20260925 01:22:27.942456 10865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } attempt: 1
I20260925 01:22:27.944156 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.943297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39092)
0925 01:22:27.943632 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.943884 (+ 252us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260925 01:22:27.945621 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.945058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39104)
0925 01:22:27.945332 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.945463 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260925 01:22:27.945922 10865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:43051: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
I20260925 01:22:27.948786 10939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } attempt: 3
I20260925 01:22:27.949803 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.949379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39106)
0925 01:22:27.949571 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.949657 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 01:22:27.950874 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:27.950485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39118)
0925 01:22:27.950703 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:27.950777 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260925 01:22:27.951149 10939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:43051: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
I20260925 01:22:28.014194 10865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } attempt: 2
I20260925 01:22:28.015430 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.014919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39120)
0925 01:22:28.015152 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:28.015268 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 01:22:28.016492 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.016147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39132)
0925 01:22:28.016317 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:28.016389 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 01:22:28.016736 10865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:43051: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
W20260925 01:22:28.084141 10946 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:22:28.084501 10946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:28.084632 10946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:28.084798 10946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:28.091818 10946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:28.091938 10946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:28.092068 10946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:28.092155 10946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:28.092247 10946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:28.094349 10939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } attempt: 4
I20260925 01:22:28.095664 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.095055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39138)
0925 01:22:28.095290 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:28.095380 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 01:22:28.096801 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.096406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39150)
0925 01:22:28.096550 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:28.096630 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 01:22:28.097245 10939 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:43051: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
I20260925 01:22:28.102629 10946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:45401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:44697,127.20.102.189:39623,127.20.102.188:43051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.188.keytab
--principal=kudu/127.20.102.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.188:43051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.102.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:28.103883 10946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:28.105330 10946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:28.117195 10951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:28.117249 10954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:28.117278 10952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:28.118083 10946 server_base.cc:1034] running on GCE node
I20260925 01:22:28.118875 10946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:28.120383 10946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:28.121671 10946 hybrid_clock.cc:648] HybridClock initialized: now 1790299348121571 us; error 74 us; skew 500 ppm
I20260925 01:22:28.122020 10946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 01:22:28.122977 10865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } attempt: 3
I20260925 01:22:28.124004 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.123559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39154)
0925 01:22:28.123754 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:28.123846 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 01:22:28.125082 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.124734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39164)
0925 01:22:28.124893 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:28.124970 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 01:22:28.125293 10865 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:43051: Network error: Client connection negotiation failed: client connection to 127.20.102.188:43051: connect: Connection refused (error 111)
Sep 25 01:22:28 dist-test-slave-s12s krb5kdc[10785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299348, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:28.130977 10946 init.cc:377] Logged in from keytab as kudu/127.20.102.188@KRBTEST.COM (short username kudu)
I20260925 01:22:28.132489 10946 webserver.cc:460] Webserver started at http://127.20.102.188:38381/ using document root <none> and password file <none>
I20260925 01:22:28.133255 10946 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:28.133347 10946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:28.133643 10946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:28.136551 10946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/data/instance:
uuid: "aced16477a81437ebfbbd9f4cd1b15d4"
format_stamp: "Formatted at 2026-09-25 01:22:28 on dist-test-slave-s12s"
server_key: "e7826fc31b03428d16689cf3ad677716"
server_key_iv: "6b28b8b80bb40e20e27cfd42adde549c"
server_key_version: "encryptionkey@0"
I20260925 01:22:28.137382 10946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/wal/instance:
uuid: "aced16477a81437ebfbbd9f4cd1b15d4"
format_stamp: "Formatted at 2026-09-25 01:22:28 on dist-test-slave-s12s"
server_key: "e7826fc31b03428d16689cf3ad677716"
server_key_iv: "6b28b8b80bb40e20e27cfd42adde549c"
server_key_version: "encryptionkey@0"
I20260925 01:22:28.142371 10946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 01:22:28.145905 10961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:28.147378 10946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:22:28.147559 10946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/wal
uuid: "aced16477a81437ebfbbd9f4cd1b15d4"
format_stamp: "Formatted at 2026-09-25 01:22:28 on dist-test-slave-s12s"
server_key: "e7826fc31b03428d16689cf3ad677716"
server_key_iv: "6b28b8b80bb40e20e27cfd42adde549c"
server_key_version: "encryptionkey@0"
I20260925 01:22:28.147794 10946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:28.169790 10946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:28.174057 10946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:28.174328 10946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:28.192536 10946 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.188:43051
I20260925 01:22:28.192572 11012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.188:43051 every 8 connection(s)
I20260925 01:22:28.193876 10946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/data/info.pb
I20260925 01:22:28.197708 11013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:28.200807 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 10946
I20260925 01:22:28.201001 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790298989998230-20890-0/minicluster-data/master-2/wal/instance
I20260925 01:22:28.201519 20890 external_mini_cluster.cc:1428] Setting key cda845e9312968a73c42b6d9874d5d3c
I20260925 01:22:28.205654 11013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:22:28 dist-test-slave-s12s krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.188@KRBTEST.COM
Sep 25 01:22:28 dist-test-slave-s12s krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299348, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.188@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:22:28.223910 10869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.208660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.20.102.190:44697)
0925 01:22:28.208848 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.208852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.209519 (+ 667us) server_negotiation.cc:407] Connection header received
0925 01:22:28.209733 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.209738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.209813 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.209921 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.211019 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.211743 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.213183 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.213618 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.213744 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.217266 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:28.217273 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:28.217278 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:28.217320 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:28.220729 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.221542 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.221549 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.221583 (+ 34us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.221728 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.222155 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.222159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.222162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.222467 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:28.222651 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:28.223323 (+ 672us) server_negotiation.cc:299] Negotiation successful
0925 01:22:28.223542 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 01:22:28.224769 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.203420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.20.102.188:43051)
0925 01:22:28.204176 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.204187 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.204225 (+ 38us) server_negotiation.cc:407] Connection header received
0925 01:22:28.206061 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.206083 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.206463 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.206813 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.208549 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.210347 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.211667 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.212342 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.212549 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.216839 (+ 4290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:28.216847 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:28.216858 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:28.216896 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:28.220422 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.221248 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.221254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.221256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.221445 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.221875 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.221878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.221880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.222539 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:28.222821 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:28.223128 (+ 307us) server_negotiation.cc:299] Negotiation successful
0925 01:22:28.223563 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":562,"spinlock_wait_cycles":13696,"thread_start_us":358,"threads_started":1}
I20260925 01:22:28.224818 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.208729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:44697 (local address 127.0.0.1:41008)
0925 01:22:28.209380 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:28.209419 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:28.209602 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:28.209991 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:28.209998 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:28.210624 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:28.210866 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:28.210874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.211902 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.211911 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:28.213043 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:28.213056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.213859 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.213867 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:28.214015 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.214636 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:28.214661 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:28.217067 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:28.220859 (+ 3792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:28.220870 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:28.220882 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:28.221415 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:28.221847 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:28.221853 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:28.221860 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:28.222029 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:28.222639 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:28.222648 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:28.222886 (+ 238us) client_negotiation.cc:769] Sending connection context
0925 01:22:28.223172 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 01:22:28.223733 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":165,"thread_start_us":344,"threads_started":1}
I20260925 01:22:28.226351 11013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:22:28 dist-test-slave-s12s krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299348, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.188@KRBTEST.COM for kudu/127.20.102.189@KRBTEST.COM
Sep 25 01:22:28 dist-test-slave-s12s krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:22:28.240320 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.227183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59374 (local address 127.20.102.189:39623)
0925 01:22:28.227394 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.227400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.227564 (+ 164us) server_negotiation.cc:407] Connection header received
0925 01:22:28.227737 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.227742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.227846 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.227971 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.228610 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.229399 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.230820 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.231240 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.231371 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.234232 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:28.234237 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:28.234240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:28.234271 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:28.237561 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.238348 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.238354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.238357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.238526 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.238962 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.238967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.238969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.239332 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:28.239502 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:28.239898 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 01:22:28.240060 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 01:22:28.240492 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.227188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:39623 (local address 127.0.0.1:59374)
0925 01:22:28.227473 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:28.227496 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:28.227620 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:28.228009 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:28.228013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:28.228346 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:28.228467 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:28.228474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.229603 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.229608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:28.230691 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:28.230703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.231378 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.231383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:28.231590 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.232104 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:28.232120 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:28.234035 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:28.237698 (+ 3663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:28.237702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:28.237704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:28.238231 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:28.238671 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:28.238675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:28.238678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:28.238838 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:28.239468 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:28.239472 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:28.239637 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 01:22:28.239758 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 01:22:28.240103 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260925 01:22:28.241797 11013 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:22:28.243147 10869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.227966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.20.102.190:44697)
0925 01:22:28.228390 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.228393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.228406 (+ 13us) server_negotiation.cc:407] Connection header received
0925 01:22:28.228441 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.228444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.228492 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.228607 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.229756 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.230455 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.231653 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.232235 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.232371 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.236316 (+ 3945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:28.236323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:28.236329 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:28.236371 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:28.239715 (+ 3344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.240642 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.240648 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.240651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.240850 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.241297 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.241302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.241305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.241719 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:28.241994 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:28.242315 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 01:22:28.242574 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337}
Sep 25 01:22:28 dist-test-slave-s12s krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299348, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.188@KRBTEST.COM for kudu/127.20.102.188@KRBTEST.COM
Sep 25 01:22:28 dist-test-slave-s12s krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.189@KRBTEST.COM
I20260925 01:22:28.255834 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.242619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39186 (local address 127.20.102.188:43051)
0925 01:22:28.242786 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.242790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.242836 (+ 46us) server_negotiation.cc:407] Connection header received
0925 01:22:28.242963 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.242968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.243032 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.243127 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.243805 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.244472 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.245717 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.246322 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.246442 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.250148 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:28.250153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:28.250156 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:28.250184 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:28.253308 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.253908 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.253915 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.253918 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.254058 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.254429 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.254434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.254438 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.254739 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:28.254898 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:28.255229 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 01:22:28.255461 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 01:22:28.255931 11017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.242590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39186)
0925 01:22:28.242752 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:28.242769 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:28.242864 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:28.243150 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:28.243154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:28.243547 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:28.243676 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:28.243685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.244595 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.244600 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:28.245578 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:28.245589 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.246583 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.246586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:28.246747 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.247293 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:28.247328 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:28.249973 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:28.253432 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:28.253438 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:28.253442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:28.253794 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:28.254188 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:28.254192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:28.254194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:28.254316 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:28.254878 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:28.254883 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:28.255010 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 01:22:28.255152 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 01:22:28.255625 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":174}
I20260925 01:22:28.261087 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.244841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.20.102.189:39623)
0925 01:22:28.245836 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.245841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.245856 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:22:28.245898 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.245902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.245968 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.246063 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.247388 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.248489 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.250344 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.251049 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.251164 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.255517 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:28.255523 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:28.255529 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:28.255567 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:28.258468 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.259270 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.259274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.259276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.259401 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.259826 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.259830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.259831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.260065 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:28.260259 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:28.260597 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 01:22:28.260815 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885}
I20260925 01:22:28.261895 11013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4: Bootstrap starting.
I20260925 01:22:28.265570 11013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:28.265887 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:28.266901 11013 log.cc:824] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:28.270130 11013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4: No bootstrap required, opened a new log
I20260925 01:22:28.269400 10840 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fd70540803a747308a92752104bc9dbd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41020
I20260925 01:22:28.280977 11013 raft_consensus.cc:348] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } }
I20260925 01:22:28.281409 11013 raft_consensus.cc:729] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aced16477a81437ebfbbd9f4cd1b15d4, State: Initialized, Role: FOLLOWER
I20260925 01:22:28.281932 11013 consensus_queue.cc:260] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } }
I20260925 01:22:28.282953 11019 sys_catalog.cc:455] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } } }
I20260925 01:22:28.283378 11019 sys_catalog.cc:458] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:22:28.284245 11013 sys_catalog.cc:564] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 01:22:28.294483 11030 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:22:28.294625 11030 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:22:28.302515 10865 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } attempt: 4
I20260925 01:22:28.292295 10946 master_runner.cc:186] Error getting master registration for 127.20.102.190:44697: OK, instance_id { permanent_uuid: "fd70540803a747308a92752104bc9dbd" instance_seqno: 1790299347154946 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:22:28.303732 10946 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 01:22:28 dist-test-slave-s12s krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for kudu/127.20.102.188@KRBTEST.COM
I20260925 01:22:28.318742 10868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.303278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39194)
0925 01:22:28.303537 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:28.303559 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:28.303695 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:28.304092 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:28.304098 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:28.304605 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:28.304861 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:28.304874 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.306077 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.306083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:28.307086 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:28.307097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.307876 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.307881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:28.308084 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.308738 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:28.308765 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:28.311662 (+ 2897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:28.315795 (+ 4133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:28.315801 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:28.315806 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:28.316343 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:28.316767 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:28.316773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:28.316777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:28.316952 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:28.317665 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:28.317670 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:28.317849 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 01:22:28.318013 (+ 164us) client_negotiation.cc:240] Negotiation successful
0925 01:22:28.318345 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260925 01:22:28.318761 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.303332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39194 (local address 127.20.102.188:43051)
0925 01:22:28.303649 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.303656 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.303677 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:22:28.303861 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.303867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.303945 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.304046 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.305012 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.305932 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.307231 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.307707 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.307851 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.311874 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:28.311882 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:28.311889 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:28.311929 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:28.315649 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.316461 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.316468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.316472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.316614 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.317071 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.317079 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.317084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.317520 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:28.317684 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:28.318119 (+ 435us) server_negotiation.cc:299] Negotiation successful
0925 01:22:28.318377 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260925 01:22:28.324867 10865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd: Bootstrap starting.
I20260925 01:22:28.328061 10865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:28.329372 10865 log.cc:824] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:28.332592 10865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd: No bootstrap required, opened a new log
I20260925 01:22:28.337718 10865 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } }
I20260925 01:22:28.338047 10865 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd70540803a747308a92752104bc9dbd, State: Initialized, Role: FOLLOWER
I20260925 01:22:28.338497 10865 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } }
I20260925 01:22:28.339502 11033 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } } }
I20260925 01:22:28.339910 11033 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:22:28.340699 10865 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:28.340760 10840 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 01:22:28.340950 10840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:28.345095 10840 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } }
I20260925 01:22:28.346922 10840 leader_election.cc:288] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [CANDIDATE]: Term 1 election: Requested vote from peers 67359673c33b4aa681b62297349cd07a (127.20.102.189:39623), aced16477a81437ebfbbd9f4cd1b15d4 (127.20.102.188:43051)
I20260925 01:22:28.348093 10988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd70540803a747308a92752104bc9dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aced16477a81437ebfbbd9f4cd1b15d4"
I20260925 01:22:28.348512 10988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:28.352790 10988 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd70540803a747308a92752104bc9dbd in term 1.
I20260925 01:22:28.353408 10817 leader_election.cc:302] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aced16477a81437ebfbbd9f4cd1b15d4, fd70540803a747308a92752104bc9dbd; no voters:
I20260925 01:22:28.353860 11033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:28.356324 11033 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [term 1 LEADER]: Becoming Leader. State: Replica: fd70540803a747308a92752104bc9dbd, State: Running, Role: LEADER
I20260925 01:22:28.356941 11033 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } }
I20260925 01:22:28.377004 11042 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd70540803a747308a92752104bc9dbd. Latest consensus state: current_term: 1 leader_uuid: "fd70540803a747308a92752104bc9dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } } }
I20260925 01:22:28.377470 11042 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:28.389302 10869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.352270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41036 (local address 127.20.102.190:44697)
0925 01:22:28.352452 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.352459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.356136 (+ 3677us) server_negotiation.cc:407] Connection header received
0925 01:22:28.357131 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.357136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.357222 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.357296 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.362722 (+ 5426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.363908 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.367833 (+ 3925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.368527 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.368628 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.380771 (+ 12143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:28.380779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:28.380785 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:28.380854 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:28.385256 (+ 4402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.386368 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.386374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.386377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.386612 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.387168 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.387177 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.387183 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.387631 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:28.387913 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:28.388550 (+ 637us) server_negotiation.cc:299] Negotiation successful
0925 01:22:28.388855 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 01:22:28.394486 10819 tablet.cc:2369] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:22:28.395757 10943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.352434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59390 (local address 127.20.102.189:39623)
0925 01:22:28.352655 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.352663 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.358222 (+ 5559us) server_negotiation.cc:407] Connection header received
0925 01:22:28.358482 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.358487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.358570 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.358647 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.362835 (+ 4188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.363926 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.367806 (+ 3880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.368524 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.368625 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.375003 (+ 6378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:28.375009 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:28.375014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:28.375060 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:28.381120 (+ 6060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.389217 (+ 8097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.389222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.389226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.389712 (+ 486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.391218 (+ 1506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.391223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.391226 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.391637 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:28.391873 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:28.393752 (+ 1879us) server_negotiation.cc:299] Negotiation successful
0925 01:22:28.393995 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 01:22:28.396428 10939 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } attempt: 5
I20260925 01:22:28.408143 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.358723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.20.102.188:43051)
0925 01:22:28.358893 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.358900 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.375145 (+ 16245us) server_negotiation.cc:407] Connection header received
0925 01:22:28.378501 (+ 3356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.378511 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.378611 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.378686 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.391423 (+ 12737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.392570 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.394668 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.395383 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.395494 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.399339 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:28.399349 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:28.399356 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:28.399409 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:28.404171 (+ 4762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.405153 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.405162 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.405168 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.405423 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.405926 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.405934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.405939 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.406333 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:28.406655 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:28.407362 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 01:22:28.407661 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 01:22:28 dist-test-slave-s12s krb5kdc[10785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299347, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.189@KRBTEST.COM for kudu/127.20.102.188@KRBTEST.COM
I20260925 01:22:28.417327 11046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260925 01:22:28.426277 10967 tablet.cc:2369] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:22:28.429272 11046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:28.433494 10941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.396997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:43051 (local address 127.0.0.1:39212)
0925 01:22:28.397366 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:28.397387 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:28.397499 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:28.408854 (+ 11355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:28.408868 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:28.409438 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:28.409707 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:28.409723 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.411499 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.411506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:28.413084 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:28.413101 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.414232 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.414238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:28.414465 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.415119 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:28.415141 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:28.417989 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:28.427744 (+ 9755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:28.427751 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:28.427755 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:28.428287 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:28.431341 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:28.431346 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:28.431350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:28.431572 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:28.432339 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:28.432344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:28.432528 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 01:22:28.432704 (+ 176us) client_negotiation.cc:240] Negotiation successful
0925 01:22:28.433042 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260925 01:22:28.434717 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:28.397855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39212 (local address 127.20.102.188:43051)
0925 01:22:28.408453 (+ 10598us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.408461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.408486 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:22:28.408542 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.408550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.408647 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.409874 (+ 1227us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.410061 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.411308 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.413291 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:28.414051 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:28.418155 (+ 4104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:28.418370 (+ 215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:28.418377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:28.418381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:28.418409 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:28.427537 (+ 9128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.428466 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.428474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.428479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.428640 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:28.431752 (+ 3112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:28.431759 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:28.431762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:28.432182 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:28.433870 (+ 1688us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:28.434009 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 01:22:28.434283 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10505,"thread_start_us":15804,"threads_started":1}
I20260925 01:22:28.441733 10939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a: Bootstrap starting.
I20260925 01:22:28.447551 10988 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4 [term 1 FOLLOWER]: Refusing update from remote peer fd70540803a747308a92752104bc9dbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:22:28.446527 10939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:28.449471 11042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:22:28.457065 10939 log.cc:824] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:28.471292 10939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a: No bootstrap required, opened a new log
I20260925 01:22:28.478307 11019 sys_catalog.cc:455] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd70540803a747308a92752104bc9dbd. Latest consensus state: current_term: 1 leader_uuid: "fd70540803a747308a92752104bc9dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } } }
I20260925 01:22:28.478662 11019 sys_catalog.cc:458] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:22:28.482782 11055 mvcc.cc:205] Tried to move back new op lower bound from 7333066131225894912 to 7333066130875879424. Current Snapshot: MvccSnapshot[applied={T|T < 7333066131225894912}]
I20260925 01:22:28.489236 11058 mvcc.cc:205] Tried to move back new op lower bound from 7333066131225894912 to 7333066130875879424. Current Snapshot: MvccSnapshot[applied={T|T < 7333066131225894912}]
I20260925 01:22:28.497810 11019 sys_catalog.cc:455] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd70540803a747308a92752104bc9dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } } }
I20260925 01:22:28.498065 11019 sys_catalog.cc:458] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:22:28.503937 10939 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } }
I20260925 01:22:28.504388 10939 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67359673c33b4aa681b62297349cd07a, State: Initialized, Role: FOLLOWER
I20260925 01:22:28.504959 10939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } }
I20260925 01:22:28.498831 11033 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd70540803a747308a92752104bc9dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } } }
I20260925 01:22:28.506631 11033 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:28.513797 11063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } } }
I20260925 01:22:28.514235 11063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:22:28.515306 10914 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd70540803a747308a92752104bc9dbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "67359673c33b4aa681b62297349cd07a"
I20260925 01:22:28.515790 10913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:28.515281 10939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:28.520972 10913 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a [term 1 FOLLOWER]: Refusing update from remote peer fd70540803a747308a92752104bc9dbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:22:28.473093 10798 master_runner.cc:186] Error getting master registration for 127.20.102.189:39623: OK, instance_id { permanent_uuid: "67359673c33b4aa681b62297349cd07a" instance_seqno: 1790299347663330 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:22:28.523118 11046 catalog_manager.cc:1347] Generated new cluster ID: 3b0ddb3c34b4489f9a26300adf8e3b95
I20260925 01:22:28.523212 11046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:28.525532 10914 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd70540803a747308a92752104bc9dbd in term 1.
I20260925 01:22:28.529996 10798 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 01:22:28.543233 11064 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:22:28.573585 11063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd70540803a747308a92752104bc9dbd. Latest consensus state: current_term: 1 leader_uuid: "fd70540803a747308a92752104bc9dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } } }
I20260925 01:22:28.574059 11063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:22:28.574497 11072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fd70540803a747308a92752104bc9dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } } }
I20260925 01:22:28.575076 11072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:22:28.622866 11064 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fd70540803a747308a92752104bc9dbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd70540803a747308a92752104bc9dbd" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 44697 } } peers { permanent_uuid: "67359673c33b4aa681b62297349cd07a" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 39623 } } peers { permanent_uuid: "aced16477a81437ebfbbd9f4cd1b15d4" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 43051 } } }
I20260925 01:22:28.623346 11064 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:28.623705 11046 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:28.625697 11046 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:28.631075 11079 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 01:22:28.646569 11046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd70540803a747308a92752104bc9dbd: Generated new TSK 0
I20260925 01:22:28.647979 11046 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 01:22:28.761214 10869 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:28.746630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41050 (local address 127.20.102.190:44697)
0925 01:22:28.746850 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.746859 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.758864 (+ 12005us) server_negotiation.cc:407] Connection header received
0925 01:22:28.759436 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.759442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.759542 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.759631 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.760941 (+ 1310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41050: BlockingRecv error: recv got EOF from 127.0.0.1:41050 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 01:22:28.763964 10943 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:28.746213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.20.102.189:39623)
0925 01:22:28.746415 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.746422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.762030 (+ 15608us) server_negotiation.cc:407] Connection header received
0925 01:22:28.762322 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.762328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.762417 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.762492 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.763744 (+ 1252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59394: BlockingRecv error: recv got EOF from 127.0.0.1:59394 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 01:22:28.764863 11016 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:28.762231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.20.102.188:43051)
0925 01:22:28.762757 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.762765 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.762785 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:22:28.762845 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.762850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.762920 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.763023 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.764604 (+ 1581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39222: BlockingRecv error: recv got EOF from 127.0.0.1:39222 (error 108)
Metrics: {"server-negotiator.queue_time_us":402}
W20260925 01:22:28.771530 10943 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:28.768420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59402 (local address 127.20.102.189:39623)
0925 01:22:28.768614 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.768621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.770179 (+ 1558us) server_negotiation.cc:407] Connection header received
0925 01:22:28.770421 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.770426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.770519 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.770640 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.771316 (+ 676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59402: BlockingRecv error: recv got EOF from 127.0.0.1:59402 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 01:22:28.778990 10943 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:28.776892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59416 (local address 127.20.102.189:39623)
0925 01:22:28.777131 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.777139 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.777627 (+ 488us) server_negotiation.cc:407] Connection header received
0925 01:22:28.777823 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.777828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.777906 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.778016 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.778793 (+ 777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59416: BlockingRecv error: recv got EOF from 127.0.0.1:59416 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 01:22:28.785907 10943 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:28.783819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.20.102.189:39623)
0925 01:22:28.784028 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.784037 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.784506 (+ 469us) server_negotiation.cc:407] Connection header received
0925 01:22:28.784742 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.784747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.784835 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.784942 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.785678 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59422: BlockingRecv error: recv got EOF from 127.0.0.1:59422 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260925 01:22:28.800792 10943 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:28.798514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.20.102.189:39623)
0925 01:22:28.798793 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.798802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.799404 (+ 602us) server_negotiation.cc:407] Connection header received
0925 01:22:28.799643 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.799647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.799714 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.799821 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.800537 (+ 716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59430: BlockingRecv error: recv got EOF from 127.0.0.1:59430 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260925 01:22:28.813864 10943 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:28.811695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59432 (local address 127.20.102.189:39623)
0925 01:22:28.811967 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.811977 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.812479 (+ 502us) server_negotiation.cc:407] Connection header received
0925 01:22:28.812737 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.812741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.812807 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.812915 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.813638 (+ 723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59432: BlockingRecv error: recv got EOF from 127.0.0.1:59432 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260925 01:22:28.835424 10943 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:28.833196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.20.102.189:39623)
0925 01:22:28.833419 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.833428 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.833977 (+ 549us) server_negotiation.cc:407] Connection header received
0925 01:22:28.834206 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.834211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.834298 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.834409 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.835159 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59438: BlockingRecv error: recv got EOF from 127.0.0.1:59438 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 01:22:28.873131 10943 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:28.870782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.20.102.189:39623)
0925 01:22:28.871045 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.871055 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.871628 (+ 573us) server_negotiation.cc:407] Connection header received
0925 01:22:28.871868 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.871873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.871949 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.872073 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.872848 (+ 775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59450: BlockingRecv error: recv got EOF from 127.0.0.1:59450 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260925 01:22:28.943039 10943 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:28.940614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59466 (local address 127.20.102.189:39623)
0925 01:22:28.940901 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:28.940908 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:28.941471 (+ 563us) server_negotiation.cc:407] Connection header received
0925 01:22:28.941704 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:28.941709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:28.941790 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:28.941912 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:28.942758 (+ 846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59466: BlockingRecv error: recv got EOF from 127.0.0.1:59466 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
W20260925 01:22:29.076423 10943 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:29.074594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.20.102.189:39623)
0925 01:22:29.074781 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:29.074785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:29.075103 (+ 318us) server_negotiation.cc:407] Connection header received
0925 01:22:29.075293 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:29.075297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:29.075374 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:29.075481 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:29.076226 (+ 745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59468: BlockingRecv error: recv got EOF from 127.0.0.1:59468 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:22:29.296890 11030 catalog_manager.cc:1259] Loaded cluster ID: 3b0ddb3c34b4489f9a26300adf8e3b95
I20260925 01:22:29.297003 11030 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4: loading cluster ID for follower catalog manager: success
I20260925 01:22:29.299726 11030 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4: acquiring CA information for follower catalog manager: success
I20260925 01:22:29.301769 11030 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aced16477a81437ebfbbd9f4cd1b15d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 01:22:29.340356 10943 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:29.338463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.20.102.189:39623)
0925 01:22:29.338677 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:29.338682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:29.338969 (+ 287us) server_negotiation.cc:407] Connection header received
0925 01:22:29.339151 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:29.339155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:29.339243 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:29.339343 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:29.340137 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59474: BlockingRecv error: recv got EOF from 127.0.0.1:59474 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 01:22:29.635526 11079 catalog_manager.cc:1259] Loaded cluster ID: 3b0ddb3c34b4489f9a26300adf8e3b95
I20260925 01:22:29.635643 11079 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a: loading cluster ID for follower catalog manager: success
I20260925 01:22:29.638196 11079 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a: acquiring CA information for follower catalog manager: success
I20260925 01:22:29.639957 11079 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 67359673c33b4aa681b62297349cd07a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:22:29.865804 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:29.858394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59480 (local address 127.20.102.189:39623)
0925 01:22:29.859251 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:29.859256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:29.859274 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:22:29.859343 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:29.859347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:29.859484 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:29.859596 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:22:29.860171 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:29.861107 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:29.863184 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:29.863640 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:29.863761 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:29.864391 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:22:29.865081 (+ 690us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:22:29.865190 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:29.865359 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 01:22:29.865427 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":745,"thread_start_us":572,"threads_started":1}
I20260925 01:22:29.879473 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:29.872249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.20.102.188:43051)
0925 01:22:29.872788 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:29.872794 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:29.873167 (+ 373us) server_negotiation.cc:407] Connection header received
0925 01:22:29.873345 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:29.873351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:29.873560 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:29.873752 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:22:29.874141 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:29.875293 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:29.876600 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:29.877293 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:29.877430 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:29.878017 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:22:29.878727 (+ 710us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:22:29.878898 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:29.879037 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 01:22:29.879149 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":256,"threads_started":1}
I20260925 01:22:29.904947 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:29.886419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.20.102.189:39623)
0925 01:22:29.887034 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:29.887043 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:29.898087 (+ 11044us) server_negotiation.cc:407] Connection header received
0925 01:22:29.898359 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:29.898366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:29.898455 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:29.898523 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:22:29.899362 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:29.900408 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:29.902589 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:29.903337 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:29.903441 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:29.904054 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:22:29.904182 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:22:29.904247 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:29.904486 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 01:22:29.904572 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448}
I20260925 01:22:29.905829 11157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:29.886682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.20.102.190:44697)
0925 01:22:29.896774 (+ 10092us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:29.896783 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:29.897673 (+ 890us) server_negotiation.cc:407] Connection header received
0925 01:22:29.897814 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:29.897820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:29.898050 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:29.898228 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:22:29.899012 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:29.900173 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:29.902547 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:29.903181 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:29.903316 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:29.904123 (+ 807us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:22:29.904628 (+ 505us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:22:29.904793 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:29.905198 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 01:22:29.905282 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9910,"thread_start_us":10106,"threads_started":1}
I20260925 01:22:29.912177 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10798
I20260925 01:22:29.913156 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:29.898351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39242 (local address 127.20.102.188:43051)
0925 01:22:29.898522 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:29.898530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:29.905390 (+ 6860us) server_negotiation.cc:407] Connection header received
0925 01:22:29.906017 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:29.906022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:29.906135 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:29.906206 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:22:29.907750 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:29.908813 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:29.910749 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:29.911393 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:29.911530 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:29.912122 (+ 592us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:22:29.912248 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:22:29.912378 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:29.912613 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 01:22:29.912738 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 01:22:29.913429 10860 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:22:30.032567 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 10798
I20260925 01:22:30.054960 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10872
I20260925 01:22:30.056134 10934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:22:30.163064 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 10872
I20260925 01:22:30.184059 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 10946
I20260925 01:22:30.185328 11008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:22:30.292913 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 10946
2026-09-25T01:22:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5049 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 01:22:30 dist-test-slave-s12s krb5kdc[11164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 01:22:30 dist-test-slave-s12s krb5kdc[11164](info): set up 2 sockets
Sep 25 01:22:30 dist-test-slave-s12s krb5kdc[11164](info): commencing operation
krb5kdc: starting...
W20260925 01:22:31.409364 20890 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/test-user.keytab.
Sep 25 01:22:31 dist-test-slave-s12s krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T01:22:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T01:22:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.20.102.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
Entry for principal kudu/127.20.102.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab.
I20260925 01:22:31.665913 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:33417
--webserver_interface=127.20.102.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:33417,127.20.102.189:42977,127.20.102.188:36559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 01:22:32.000092 11177 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:22:32.000490 11177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:32.000622 11177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:32.000777 11177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:32.008116 11177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:32.008260 11177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:32.008397 11177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:32.008488 11177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:32.008605 11177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:32.018543 11177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:33417,127.20.102.189:42977,127.20.102.188:36559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.190.keytab
--principal=kudu/127.20.102.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.190:33417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.102.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:32.019860 11177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:32.021302 11177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:32.033420 11182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:32.033392 11183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:32.033435 11185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:32.034330 11177 server_base.cc:1034] running on GCE node
I20260925 01:22:32.035138 11177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:32.036552 11177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:32.037786 11177 hybrid_clock.cc:648] HybridClock initialized: now 1790299352037718 us; error 34 us; skew 500 ppm
I20260925 01:22:32.038128 11177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:32 dist-test-slave-s12s krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299352, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:32.041366 11177 init.cc:377] Logged in from keytab as kudu/127.20.102.190@KRBTEST.COM (short username kudu)
I20260925 01:22:32.042752 11177 webserver.cc:460] Webserver started at http://127.20.102.190:33589/ using document root <none> and password file <none>
I20260925 01:22:32.043390 11177 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:32.043478 11177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:32.043746 11177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:32.046689 11177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/data/instance:
uuid: "0776c251dffa401b9e3d74083e3d53e2"
format_stamp: "Formatted at 2026-09-25 01:22:32 on dist-test-slave-s12s"
server_key: "de68ac977dc3e2a29266d9d738c15d2b"
server_key_iv: "a2b1eb84349256bbf7370346de13ccb8"
server_key_version: "encryptionkey@0"
I20260925 01:22:32.047429 11177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/wal/instance:
uuid: "0776c251dffa401b9e3d74083e3d53e2"
format_stamp: "Formatted at 2026-09-25 01:22:32 on dist-test-slave-s12s"
server_key: "de68ac977dc3e2a29266d9d738c15d2b"
server_key_iv: "a2b1eb84349256bbf7370346de13ccb8"
server_key_version: "encryptionkey@0"
I20260925 01:22:32.052654 11177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 01:22:32.056298 11192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:32.057809 11177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 01:22:32.057958 11177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/wal
uuid: "0776c251dffa401b9e3d74083e3d53e2"
format_stamp: "Formatted at 2026-09-25 01:22:32 on dist-test-slave-s12s"
server_key: "de68ac977dc3e2a29266d9d738c15d2b"
server_key_iv: "a2b1eb84349256bbf7370346de13ccb8"
server_key_version: "encryptionkey@0"
I20260925 01:22:32.058140 11177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:32.087387 11177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:32.091948 11177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:32.092236 11177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:32.111425 11177 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.190:33417
I20260925 01:22:32.111438 11243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.190:33417 every 8 connection(s)
I20260925 01:22:32.112812 11177 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/data/info.pb
I20260925 01:22:32.116617 11244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:32.119567 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 11177
I20260925 01:22:32.119776 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-0/wal/instance
I20260925 01:22:32.120223 20890 external_mini_cluster.cc:1428] Setting key f44286bd57e9c888b84cf3fd12eb7701
I20260925 01:22:32.123416 11244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:22:32 dist-test-slave-s12s krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299352, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.102.189@KRBTEST.COM; defaulting to no policy
I20260925 01:22:32.148955 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.126670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33417 (local address 127.0.0.1:37982)
0925 01:22:32.127729 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.127796 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:32.129722 (+ 1926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:32.130729 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:32.130737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:32.131434 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:32.131780 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:32.131797 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.134149 (+ 2352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.134153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:32.135285 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:32.135297 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.136627 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.136632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:32.136821 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:32.137538 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:32.137565 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:32.140028 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:32.144666 (+ 4638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:32.144718 (+ 52us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:32.144737 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:32.145156 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:32.145636 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:32.145639 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:32.145641 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:32.145798 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:32.146873 (+ 1075us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:32.146879 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:32.147078 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 01:22:32.147288 (+ 210us) client_negotiation.cc:240] Negotiation successful
0925 01:22:32.147796 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":413,"threads_started":1}
I20260925 01:22:32.148993 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.126656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37982 (local address 127.20.102.190:33417)
0925 01:22:32.128342 (+ 1686us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:32.128352 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:32.128407 (+ 55us) server_negotiation.cc:407] Connection header received
0925 01:22:32.130082 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:32.130092 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:32.130410 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:32.130623 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:32.131997 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.133975 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.135430 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.136402 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.136591 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:32.140307 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:32.140322 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:32.140342 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:32.140387 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:32.144520 (+ 4133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:32.145298 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:32.145310 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:32.145315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:32.145507 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:32.145929 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:32.145935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:32.145939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:32.146719 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:32.146947 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:32.147497 (+ 550us) server_negotiation.cc:299] Negotiation successful
0925 01:22:32.147964 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":156,"server-negotiator.queue_time_us":1474,"spinlock_wait_cycles":62848,"thread_start_us":1284,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.189@KRBTEST.COM" created.
I20260925 01:22:32.151932 11244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:22:32.153034 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.152613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42977 (local address 127.0.0.1:45416)
0925 01:22:32.152835 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.152936 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260925 01:22:32.153278 11194 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
I20260925 01:22:32.154817 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.154470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42977 (local address 127.0.0.1:45432)
0925 01:22:32.154619 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.154704 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 01:22:32.154994 11244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:42977: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.189.keytab.
Entry for principal kudu/127.20.102.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.189.keytab.
I20260925 01:22:32.181509 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.189:42977
--webserver_interface=127.20.102.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.189.keytab
--principal=kudu/127.20.102.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:33417,127.20.102.189:42977,127.20.102.188:36559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:22:32.204304 11244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } attempt: 1
I20260925 01:22:32.205668 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.205184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42977 (local address 127.0.0.1:45434)
0925 01:22:32.205407 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.205527 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 01:22:32.206720 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.206343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42977 (local address 127.0.0.1:45448)
0925 01:22:32.206503 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.206576 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 01:22:32.206895 11244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:42977: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
I20260925 01:22:32.275151 11244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } attempt: 2
I20260925 01:22:32.276503 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.275981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42977 (local address 127.0.0.1:45460)
0925 01:22:32.276262 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.276358 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 01:22:32.277598 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.277225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42977 (local address 127.0.0.1:45474)
0925 01:22:32.277372 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.277470 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 01:22:32.277774 11244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:42977: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
I20260925 01:22:32.369023 11244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } attempt: 3
I20260925 01:22:32.370223 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.369786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42977 (local address 127.0.0.1:45480)
0925 01:22:32.369983 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.370076 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 01:22:32.371286 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.370911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42977 (local address 127.0.0.1:45488)
0925 01:22:32.371066 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.371140 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 01:22:32.371459 11244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:42977: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
I20260925 01:22:32.514689 11244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } attempt: 4
W20260925 01:22:32.515025 11251 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:22:32.515547 11251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:32.515669 11251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:32.515805 11251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 01:22:32.516085 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.515468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42977 (local address 127.0.0.1:45504)
0925 01:22:32.515797 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.515954 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260925 01:22:32.517266 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.516804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42977 (local address 127.0.0.1:45512)
0925 01:22:32.517003 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.517125 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260925 01:22:32.517581 11244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.189:42977: Network error: Client connection negotiation failed: client connection to 127.20.102.189:42977: connect: Connection refused (error 111)
W20260925 01:22:32.523643 11251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:32.523761 11251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:32.523886 11251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:32.523972 11251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:32.524065 11251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:32.534474 11251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:33417,127.20.102.189:42977,127.20.102.188:36559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.189.keytab
--principal=kudu/127.20.102.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.189:42977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.20.102.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:32.535743 11251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:32.537068 11251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 01:22:32.548722 11259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:32.548643 11257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:32.548637 11256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:32.549867 11251 server_base.cc:1034] running on GCE node
I20260925 01:22:32.550712 11251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:32.552119 11251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:32.553335 11251 hybrid_clock.cc:648] HybridClock initialized: now 1790299352553275 us; error 38 us; skew 500 ppm
I20260925 01:22:32.553704 11251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:32 dist-test-slave-s12s krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299352, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:32.557143 11251 init.cc:377] Logged in from keytab as kudu/127.20.102.189@KRBTEST.COM (short username kudu)
I20260925 01:22:32.558605 11251 webserver.cc:460] Webserver started at http://127.20.102.189:40781/ using document root <none> and password file <none>
I20260925 01:22:32.559316 11251 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:32.559448 11251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:32.559774 11251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:32.563112 11251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/data/instance:
uuid: "9b90e6daf4b64d17ab1354c7297bae86"
format_stamp: "Formatted at 2026-09-25 01:22:32 on dist-test-slave-s12s"
server_key: "dadcafa6fe9774711959316c2e1754d7"
server_key_iv: "4b42afc5b79be1ed1be4d6490db6b30f"
server_key_version: "encryptionkey@0"
I20260925 01:22:32.563927 11251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/wal/instance:
uuid: "9b90e6daf4b64d17ab1354c7297bae86"
format_stamp: "Formatted at 2026-09-25 01:22:32 on dist-test-slave-s12s"
server_key: "dadcafa6fe9774711959316c2e1754d7"
server_key_iv: "4b42afc5b79be1ed1be4d6490db6b30f"
server_key_version: "encryptionkey@0"
I20260925 01:22:32.569082 11251 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 01:22:32.572374 11266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:32.573753 11251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 01:22:32.573941 11251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/wal
uuid: "9b90e6daf4b64d17ab1354c7297bae86"
format_stamp: "Formatted at 2026-09-25 01:22:32 on dist-test-slave-s12s"
server_key: "dadcafa6fe9774711959316c2e1754d7"
server_key_iv: "4b42afc5b79be1ed1be4d6490db6b30f"
server_key_version: "encryptionkey@0"
I20260925 01:22:32.574158 11251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:32.586807 11251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:32.591209 11251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:32.591491 11251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:32.610306 11251 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.189:42977
I20260925 01:22:32.610329 11317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.189:42977 every 8 connection(s)
I20260925 01:22:32.611717 11251 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/data/info.pb
I20260925 01:22:32.615718 11318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:32.615983 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 11251
I20260925 01:22:32.616207 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-1/wal/instance
I20260925 01:22:32.616796 20890 external_mini_cluster.cc:1428] Setting key f0f6858cd4bd5e5b33731b46043d7efd
I20260925 01:22:32.626291 11318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:22:32 dist-test-slave-s12s krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299352, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.189@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.20.102.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.20.102.188@KRBTEST.COM" created.
I20260925 01:22:32.648313 11247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.629921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37994 (local address 127.20.102.190:33417)
0925 01:22:32.630126 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:32.630130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:32.631541 (+ 1411us) server_negotiation.cc:407] Connection header received
0925 01:22:32.633701 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:32.633707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:32.633817 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:32.633935 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:32.635526 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.636216 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.637799 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.638230 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.638372 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:32.642022 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:32.642028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:32.642031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:32.642065 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:32.644844 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:32.645607 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:32.645615 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:32.645619 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:32.645776 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:32.646235 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:32.646240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:32.646243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:32.646544 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:32.646712 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:32.647648 (+ 936us) server_negotiation.cc:299] Negotiation successful
0925 01:22:32.647930 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 01:22:32.649210 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.630510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33417 (local address 127.0.0.1:37994)
0925 01:22:32.631347 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.631393 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:32.633409 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:32.634188 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:32.634197 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:32.634752 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:32.635336 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:32.635355 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.636385 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.636389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:32.637647 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:32.637660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.638468 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.638475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:32.638722 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:32.639426 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:32.639453 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:32.641783 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:32.644990 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:32.644999 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:32.645016 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:32.645464 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:32.645909 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:32.645914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:32.645918 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:32.646113 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:32.646726 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:32.646736 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:32.647171 (+ 435us) client_negotiation.cc:769] Sending connection context
0925 01:22:32.647522 (+ 351us) client_negotiation.cc:240] Negotiation successful
0925 01:22:32.647982 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":358,"threads_started":1}
I20260925 01:22:32.651129 11318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:22:32 dist-test-slave-s12s krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299352, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.189@KRBTEST.COM for kudu/127.20.102.189@KRBTEST.COM
I20260925 01:22:32.667830 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.652023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42977 (local address 127.0.0.1:45520)
0925 01:22:32.652232 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.652253 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:32.652367 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:32.653168 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:32.653179 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:32.653710 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:32.653936 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:32.653951 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.655413 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.655421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:32.656896 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:32.656908 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.658151 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.658155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:32.658278 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:32.658897 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:32.658913 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:32.660951 (+ 2038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:32.665379 (+ 4428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:32.665382 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:32.665385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:32.665747 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:32.666169 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:32.666172 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:32.666174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:32.666295 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:32.667055 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:32.667059 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:32.667186 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 01:22:32.667314 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 01:22:32.667528 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 01:22:32.668469 11322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.652061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45520 (local address 127.20.102.189:42977)
0925 01:22:32.652757 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:32.652762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:32.652789 (+ 27us) server_negotiation.cc:407] Connection header received
0925 01:22:32.652831 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:32.652836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:32.653010 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:32.653124 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:32.654138 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.655260 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.657031 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.658005 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.658150 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:32.661166 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:32.661203 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:32.661221 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:32.661268 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:32.665238 (+ 3970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:32.665877 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:32.665887 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:32.665893 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:32.666044 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:32.666406 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:32.666414 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:32.666420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:32.666900 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:32.667105 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:32.667522 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 01:22:32.667810 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":344,"threads_started":1}
I20260925 01:22:32.670919 11318 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:22:32.672029 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.671572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41316)
0925 01:22:32.671756 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.671872 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260925 01:22:32.672401 11268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
I20260925 01:22:32.674013 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.673684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41320)
0925 01:22:32.673843 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.673918 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 01:22:32.674294 11318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:36559: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.20.102.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.188.keytab.
Entry for principal kudu/127.20.102.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.188.keytab.
I20260925 01:22:32.683218 20890 external_mini_cluster.cc:1326] Running /tmp/dist-test-task_TC7zy/build/asan/bin/kudu
/tmp/dist-test-task_TC7zy/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.188:36559
--webserver_interface=127.20.102.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.188.keytab
--principal=kudu/127.20.102.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.20.102.148:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.102.190:33417,127.20.102.189:42977,127.20.102.188:36559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 01:22:32.723615 11318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } attempt: 1
I20260925 01:22:32.725553 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.724383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41324)
0925 01:22:32.725291 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.725392 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":775}
I20260925 01:22:32.726578 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.726225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41336)
0925 01:22:32.726396 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.726475 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 01:22:32.726882 11318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:36559: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
I20260925 01:22:32.795150 11318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } attempt: 2
I20260925 01:22:32.796609 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.795999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41348)
0925 01:22:32.796249 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.796378 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 01:22:32.797809 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.797352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41358)
0925 01:22:32.797527 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.797643 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 01:22:32.798125 11318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:36559: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
I20260925 01:22:32.816814 11244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } attempt: 5
Sep 25 01:22:32 dist-test-slave-s12s krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299352, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for kudu/127.20.102.189@KRBTEST.COM
I20260925 01:22:32.831914 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.817549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42977 (local address 127.0.0.1:45524)
0925 01:22:32.817771 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.817799 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:32.817914 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:32.818314 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:32.818318 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:32.818727 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:32.818930 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:32.818941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.820388 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.820392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:32.821379 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:32.821394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.822364 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.822368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:32.822569 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:32.823273 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:32.823292 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:32.825447 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:32.829530 (+ 4083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:32.829563 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:32.829566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:32.829950 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:32.830342 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:32.830345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:32.830348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:32.830474 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:32.831134 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:32.831145 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:32.831299 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 01:22:32.831424 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 01:22:32.831650 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 01:22:32.832100 11322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.817698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45524 (local address 127.20.102.189:42977)
0925 01:22:32.817888 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:32.817894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:32.817918 (+ 24us) server_negotiation.cc:407] Connection header received
0925 01:22:32.818017 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:32.818022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:32.818122 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:32.818235 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:32.819113 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.820229 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.821536 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:32.822208 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:32.822337 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:32.825684 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:32.825694 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:32.825701 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:32.825777 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:32.829375 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:32.830071 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:32.830077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:32.830080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:32.830231 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:32.830577 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:32.830582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:32.830585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:32.830946 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:32.831097 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:32.831555 (+ 458us) server_negotiation.cc:299] Negotiation successful
0925 01:22:32.831793 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 01:22:32.833261 11244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:22:32.834311 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.833937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41374)
0925 01:22:32.834108 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.834181 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 01:22:32.835421 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.834990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41384)
0925 01:22:32.835247 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.835328 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260925 01:22:32.835678 11244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:36559: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
I20260925 01:22:32.886957 11244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } attempt: 1
I20260925 01:22:32.888320 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.887825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41390)
0925 01:22:32.888062 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.888150 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 01:22:32.889436 11318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } attempt: 3
I20260925 01:22:32.889775 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.889307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41392)
0925 01:22:32.889526 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.889626 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260925 01:22:32.890030 11244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:36559: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
I20260925 01:22:32.890480 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.890044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41400)
0925 01:22:32.890259 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.890344 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 01:22:32.891552 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.891148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41408)
0925 01:22:32.891368 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.891434 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
W20260925 01:22:32.891835 11318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:36559: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
I20260925 01:22:32.958292 11244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } attempt: 2
I20260925 01:22:32.959877 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.959164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41414)
0925 01:22:32.959405 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.959621 (+ 216us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260925 01:22:32.961277 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:32.960812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41430)
0925 01:22:32.961039 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:32.961141 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 01:22:32.961550 11244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:36559: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
I20260925 01:22:33.035037 11318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } attempt: 4
I20260925 01:22:33.036396 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.035791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41446)
0925 01:22:33.036012 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:33.036155 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 01:22:33.037494 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.037148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41448)
0925 01:22:33.037325 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:33.037400 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260925 01:22:33.037725 11318 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:36559: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
W20260925 01:22:33.040050 11325 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 01:22:33.040380 11325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 01:22:33.040513 11325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 01:22:33.040656 11325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 01:22:33.047799 11325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 01:22:33.047914 11325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 01:22:33.048036 11325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 01:22:33.048125 11325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 01:22:33.048229 11325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 01:22:33.058260 11325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.102.148:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.20.102.190:33417,127.20.102.189:42977,127.20.102.188:36559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/krb5kdc/kudu_127.20.102.188.keytab
--principal=kudu/127.20.102.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.102.188:36559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.20.102.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 00:11:47 UTC on ccycloud.quasar-qxzlch.root.comops.site
build id 4671449
ASAN enabled
I20260925 01:22:33.059558 11325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 01:22:33.061138 11325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 01:22:33.067852 11244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } attempt: 3
I20260925 01:22:33.069016 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.068425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41462)
0925 01:22:33.068668 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:33.068794 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 01:22:33.070183 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.069782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41474)
0925 01:22:33.069961 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:33.070047 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 01:22:33.070423 11244 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.20.102.188:36559: Network error: Client connection negotiation failed: client connection to 127.20.102.188:36559: connect: Connection refused (error 111)
W20260925 01:22:33.073872 11331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:33.073805 11333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 01:22:33.074388 11330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 01:22:33.074718 11325 server_base.cc:1034] running on GCE node
I20260925 01:22:33.075480 11325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 01:22:33.076880 11325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 01:22:33.078117 11325 hybrid_clock.cc:648] HybridClock initialized: now 1790299353078034 us; error 51 us; skew 500 ppm
I20260925 01:22:33.078455 11325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 01:22:33 dist-test-slave-s12s krb5kdc[11164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299353, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 01:22:33.081645 11325 init.cc:377] Logged in from keytab as kudu/127.20.102.188@KRBTEST.COM (short username kudu)
I20260925 01:22:33.083113 11325 webserver.cc:460] Webserver started at http://127.20.102.188:44691/ using document root <none> and password file <none>
I20260925 01:22:33.083843 11325 fs_manager.cc:359] Metadata directory not provided
I20260925 01:22:33.083978 11325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 01:22:33.084307 11325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 01:22:33.087308 11325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/data/instance:
uuid: "e65c16a64b2d49039193a4ef47c2d075"
format_stamp: "Formatted at 2026-09-25 01:22:33 on dist-test-slave-s12s"
server_key: "cce1bd755b673a1927b8782cb0e70442"
server_key_iv: "c5a1a0badc6f2f9305e145cce4959d52"
server_key_version: "encryptionkey@0"
I20260925 01:22:33.088126 11325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/wal/instance:
uuid: "e65c16a64b2d49039193a4ef47c2d075"
format_stamp: "Formatted at 2026-09-25 01:22:33 on dist-test-slave-s12s"
server_key: "cce1bd755b673a1927b8782cb0e70442"
server_key_iv: "c5a1a0badc6f2f9305e145cce4959d52"
server_key_version: "encryptionkey@0"
I20260925 01:22:33.093302 11325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260925 01:22:33.096895 11340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 01:22:33.098196 11325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 01:22:33.098379 11325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/data,/tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/wal
uuid: "e65c16a64b2d49039193a4ef47c2d075"
format_stamp: "Formatted at 2026-09-25 01:22:33 on dist-test-slave-s12s"
server_key: "cce1bd755b673a1927b8782cb0e70442"
server_key_iv: "c5a1a0badc6f2f9305e145cce4959d52"
server_key_version: "encryptionkey@0"
I20260925 01:22:33.098590 11325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 01:22:33.120580 11325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 01:22:33.124811 11325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 01:22:33.125022 11325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 01:22:33.143113 11325 rpc_server.cc:307] RPC server started. Bound to: 127.20.102.188:36559
I20260925 01:22:33.143126 11391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.20.102.188:36559 every 8 connection(s)
I20260925 01:22:33.144445 11325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/data/info.pb
I20260925 01:22:33.148104 11392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 01:22:33.151680 20890 external_mini_cluster.cc:1388] Started /tmp/dist-test-task_TC7zy/build/asan/bin/kudu as pid 11325
I20260925 01:22:33.151865 20890 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task_TC7zy/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790298989998230-20890-0/minicluster-data/master-2/wal/instance
I20260925 01:22:33.152417 20890 external_mini_cluster.cc:1428] Setting key e6cb975f714d10330d9252069acd2e68
I20260925 01:22:33.155838 11392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:22:33 dist-test-slave-s12s krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299353, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.188@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
Sep 25 01:22:33 dist-test-slave-s12s krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.188@KRBTEST.COM
I20260925 01:22:33.179081 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.159410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37998 (local address 127.20.102.190:33417)
0925 01:22:33.159998 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.160004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.160532 (+ 528us) server_negotiation.cc:407] Connection header received
0925 01:22:33.160737 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.160743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.160820 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.160922 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.162059 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.163074 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.164719 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.165169 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.165276 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.172324 (+ 7048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:33.172334 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:33.172352 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:33.172397 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:33.175374 (+ 2977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.176247 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.176251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.176254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.176392 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.176922 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.176927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.176931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.177347 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:33.177526 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:33.178406 (+ 880us) server_negotiation.cc:299] Negotiation successful
0925 01:22:33.178646 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":315,"threads_started":1}
I20260925 01:22:33.180207 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.159488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.190:33417 (local address 127.0.0.1:37998)
0925 01:22:33.160379 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:33.160426 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:33.160580 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:33.161019 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:33.161023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:33.161613 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:33.161909 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:33.161919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.163223 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.163230 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:33.164521 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:33.164537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.165354 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.165359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:33.165476 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.169786 (+ 4310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:33.169808 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:33.172070 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:33.175516 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:33.175529 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:33.175547 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:33.176104 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:33.176528 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:33.176532 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:33.176536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:33.176801 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:33.177502 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:33.177511 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:33.177941 (+ 430us) client_negotiation.cc:769] Sending connection context
0925 01:22:33.178274 (+ 333us) client_negotiation.cc:240] Negotiation successful
0925 01:22:33.178801 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":495,"threads_started":1}
I20260925 01:22:33.180233 11395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.154331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.20.102.188:36559)
0925 01:22:33.155062 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.155073 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.155114 (+ 41us) server_negotiation.cc:407] Connection header received
0925 01:22:33.156671 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.156749 (+ 78us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.157157 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.157603 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.159309 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.161109 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.162700 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.163647 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.163819 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.173420 (+ 9601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:33.173432 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:33.173449 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:33.173497 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:33.176644 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.177502 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.177511 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.177514 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.177666 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.178022 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.178029 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.178032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.178523 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:33.178808 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:33.179168 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 01:22:33.179407 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"spinlock_wait_cycles":18304,"thread_start_us":337,"threads_started":1}
I20260925 01:22:33.181941 11392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 01:22:33 dist-test-slave-s12s krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299353, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.188@KRBTEST.COM for kudu/127.20.102.189@KRBTEST.COM
Sep 25 01:22:33 dist-test-slave-s12s krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.190@KRBTEST.COM
I20260925 01:22:33.195480 11322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.182751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45540 (local address 127.20.102.189:42977)
0925 01:22:33.182921 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.182927 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.183155 (+ 228us) server_negotiation.cc:407] Connection header received
0925 01:22:33.183293 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.183301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.183389 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.183481 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.184137 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.184891 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.186031 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.186473 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.186615 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.189501 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:33.189508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:33.189513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:33.189556 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:33.192747 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.193515 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.193522 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.193527 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.193693 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.194149 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.194153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.194155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.194399 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:33.194585 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:33.195048 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 01:22:33.195214 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 01:22:33.195595 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.182838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.189:42977 (local address 127.0.0.1:45540)
0925 01:22:33.183053 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:33.183066 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:33.183158 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:33.183502 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:33.183506 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:33.183854 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:33.183989 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:33.183997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.185021 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.185028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:33.185886 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:33.185897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.186622 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.186628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:33.186804 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.187330 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:33.187346 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:33.189305 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:33.192871 (+ 3566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:33.192876 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:33.192879 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:33.193398 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:33.193819 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:33.193824 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:33.193827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:33.194012 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:33.194555 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:33.194560 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:33.194781 (+ 221us) client_negotiation.cc:769] Sending connection context
0925 01:22:33.194949 (+ 168us) client_negotiation.cc:240] Negotiation successful
0925 01:22:33.195255 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260925 01:22:33.196622 11392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } has no permanent_uuid. Determining permanent_uuid...
I20260925 01:22:33.199117 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.183499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38010 (local address 127.20.102.190:33417)
0925 01:22:33.183688 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.183693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.183760 (+ 67us) server_negotiation.cc:407] Connection header received
0925 01:22:33.183957 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.183960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.184027 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.184135 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.185319 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.185980 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.187161 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.187576 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.187704 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.192842 (+ 5138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:33.192850 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:33.192854 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:33.192892 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:33.196264 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.197056 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.197064 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.197067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.197203 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.197646 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.197650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.197653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.198032 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:33.198327 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:33.198558 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 01:22:33.198788 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 01:22:33 dist-test-slave-s12s krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299353, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.188@KRBTEST.COM for kudu/127.20.102.188@KRBTEST.COM
Sep 25 01:22:33 dist-test-slave-s12s krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.20.102.189@KRBTEST.COM
I20260925 01:22:33.210861 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.197344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41486)
0925 01:22:33.197517 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:33.197536 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:33.197638 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:33.197904 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:33.197909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:33.198375 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:33.198560 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:33.198572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.199774 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.199778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:33.200991 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:33.201006 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.201923 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.201927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:33.202175 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.202739 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:33.202754 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:33.205214 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:33.208222 (+ 3008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:33.208227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:33.208231 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:33.208768 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:33.209159 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:33.209162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:33.209164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:33.209283 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:33.209832 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:33.209836 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:33.209959 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 01:22:33.210151 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 01:22:33.210457 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":21}
I20260925 01:22:33.210896 11395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.197505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41486 (local address 127.20.102.188:36559)
0925 01:22:33.197649 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.197653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.197668 (+ 15us) server_negotiation.cc:407] Connection header received
0925 01:22:33.197727 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.197730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.197779 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.197848 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.198713 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.199634 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.201137 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.201759 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.201914 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.205402 (+ 3488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:33.205410 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:33.205418 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:33.205459 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:33.208080 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.208892 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.208897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.208900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.209037 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.209428 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.209433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.209437 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.209702 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:33.209812 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:33.210255 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 01:22:33.210590 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":186,"server-negotiator.queue_time_us":47}
I20260925 01:22:33.215740 11392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075: Bootstrap starting.
I20260925 01:22:33.216416 11322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.200861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.20.102.189:42977)
0925 01:22:33.201051 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.201059 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.201077 (+ 18us) server_negotiation.cc:407] Connection header received
0925 01:22:33.201128 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.201132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.201202 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.201269 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.202903 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.204013 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.205500 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.205888 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.206019 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.210676 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:33.210681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:33.210684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:33.210713 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:33.213882 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.214527 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.214532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.214534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.214678 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.215075 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.215079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.215081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.215348 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:33.215645 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:33.215899 (+ 254us) server_negotiation.cc:299] Negotiation successful
0925 01:22:33.216121 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 01:22:33.219738 11392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:33.221192 20890 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 01:22:33.221151 11392 log.cc:824] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:33.224150 11392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075: No bootstrap required, opened a new log
I20260925 01:22:33.224082 11219 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0776c251dffa401b9e3d74083e3d53e2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38010
I20260925 01:22:33.234874 11392 raft_consensus.cc:348] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } }
I20260925 01:22:33.235292 11392 raft_consensus.cc:729] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e65c16a64b2d49039193a4ef47c2d075, State: Initialized, Role: FOLLOWER
I20260925 01:22:33.235793 11392 consensus_queue.cc:260] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } }
I20260925 01:22:33.236765 11399 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } } }
I20260925 01:22:33.237115 11399 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:22:33.237963 11392 sys_catalog.cc:564] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:33.247570 11244 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } attempt: 4
W20260925 01:22:33.247790 11410 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:22:33.250159 11410 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:22:33.245671 11325 master_runner.cc:186] Error getting master registration for 127.20.102.190:33417: OK, instance_id { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" instance_seqno: 1790299352106024 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:22:33.257087 11325 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 25 01:22:33 dist-test-slave-s12s krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299352, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.190@KRBTEST.COM for kudu/127.20.102.188@KRBTEST.COM
I20260925 01:22:33.266121 11246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.248156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41492)
0925 01:22:33.248410 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:33.248429 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:33.248555 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:33.249144 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:33.249149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:33.249674 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:33.249882 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:33.249895 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.251453 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.251459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:33.252996 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:33.253012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.256062 (+ 3050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.256069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:33.256302 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.257054 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:33.257091 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:33.259540 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:33.263212 (+ 3672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:33.263217 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:33.263222 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:33.263720 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:33.264140 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:33.264145 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:33.264148 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:33.264354 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:33.265119 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:33.265124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:33.265296 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 01:22:33.265445 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 01:22:33.265718 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260925 01:22:33.266197 11395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.248230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41492 (local address 127.20.102.188:36559)
0925 01:22:33.248413 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.248424 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.248494 (+ 70us) server_negotiation.cc:407] Connection header received
0925 01:22:33.248758 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.248763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.248852 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.248959 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.250017 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.251279 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.253135 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.253941 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.254080 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.259728 (+ 5648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:33.259733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:33.259738 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:33.259776 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:33.263088 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.263865 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.263871 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.263875 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.264023 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.264468 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.264473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.264476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.264985 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:33.265160 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:33.265582 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 01:22:33.265824 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 01:22:33.271837 11244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2: Bootstrap starting.
I20260925 01:22:33.275043 11244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:33.276109 11244 log.cc:824] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:33.279582 11244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2: No bootstrap required, opened a new log
I20260925 01:22:33.285072 11244 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } }
I20260925 01:22:33.285439 11244 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0776c251dffa401b9e3d74083e3d53e2, State: Initialized, Role: FOLLOWER
I20260925 01:22:33.285892 11244 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } }
I20260925 01:22:33.286842 11413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } } }
I20260925 01:22:33.287261 11413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:22:33.288036 11244 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:33.288209 11219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 01:22:33.288453 11219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:33.292258 11219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } }
I20260925 01:22:33.295176 11367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0776c251dffa401b9e3d74083e3d53e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e65c16a64b2d49039193a4ef47c2d075"
I20260925 01:22:33.295717 11367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:33.296926 11219 leader_election.cc:288] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [CANDIDATE]: Term 1 election: Requested vote from peers 9b90e6daf4b64d17ab1354c7297bae86 (127.20.102.189:42977), e65c16a64b2d49039193a4ef47c2d075 (127.20.102.188:36559)
I20260925 01:22:33.301103 11367 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776c251dffa401b9e3d74083e3d53e2 in term 1.
I20260925 01:22:33.304054 11194 leader_election.cc:302] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0776c251dffa401b9e3d74083e3d53e2, e65c16a64b2d49039193a4ef47c2d075; no voters:
I20260925 01:22:33.304636 11413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 01:22:33.312970 11413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [term 1 LEADER]: Becoming Leader. State: Replica: 0776c251dffa401b9e3d74083e3d53e2, State: Running, Role: LEADER
I20260925 01:22:33.313692 11413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } }
W20260925 01:22:33.316120 11424 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 01:22:33.316303 11424 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 01:22:33.305294 11177 master_runner.cc:186] Error getting master registration for 127.20.102.189:42977: OK, instance_id { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" instance_seqno: 1790299352604888 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 01:22:33.323338 11177 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 01:22:33.330156 11427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0776c251dffa401b9e3d74083e3d53e2. Latest consensus state: current_term: 1 leader_uuid: "0776c251dffa401b9e3d74083e3d53e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } } }
I20260925 01:22:33.330643 11427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:33.336931 11318 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } attempt: 5
I20260925 01:22:33.338289 11434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260925 01:22:33.345256 11198 tablet.cc:2369] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:22:33.351393 11434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 01:22:33.362828 11322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.327666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.20.102.189:42977)
0925 01:22:33.327873 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.327880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.333931 (+ 6051us) server_negotiation.cc:407] Connection header received
0925 01:22:33.334929 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.334938 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.335064 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.335155 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.339130 (+ 3975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.340262 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.342296 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.342978 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.343084 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.353852 (+ 10768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:33.353860 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:33.353865 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:33.353908 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:33.357654 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.358680 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.358686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.358690 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.358869 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.359430 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.359436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.359439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.361263 (+ 1824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:33.361497 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:33.362119 (+ 622us) server_negotiation.cc:299] Negotiation successful
0925 01:22:33.362389 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":77,"server-negotiator.queue_time_us":75}
Sep 25 01:22:33 dist-test-slave-s12s krb5kdc[11164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790299352, etypes {rep=17 tkt=17 ses=17}, kudu/127.20.102.189@KRBTEST.COM for kudu/127.20.102.188@KRBTEST.COM
I20260925 01:22:33.373411 11321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.337487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.20.102.188:36559 (local address 127.0.0.1:41512)
0925 01:22:33.337867 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 01:22:33.337886 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 01:22:33.337998 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 01:22:33.351571 (+ 13573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 01:22:33.351576 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 01:22:33.352091 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 01:22:33.352323 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:33.352336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.359593 (+ 7257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.359598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:33.361109 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 01:22:33.361124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.362211 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.362217 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 01:22:33.362458 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.363026 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 01:22:33.363049 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 01:22:33.365670 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 01:22:33.370521 (+ 4851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:33.370529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:33.370534 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:33.371020 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:33.371455 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 01:22:33.371461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 01:22:33.371465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 01:22:33.371666 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 01:22:33.372427 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 01:22:33.372432 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 01:22:33.372578 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 01:22:33.372760 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 01:22:33.373044 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":29}
I20260925 01:22:33.373960 11395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.332963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.20.102.188:36559)
0925 01:22:33.333148 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.333154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.345312 (+ 12158us) server_negotiation.cc:407] Connection header received
0925 01:22:33.345737 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.345743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.345846 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.345918 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.351810 (+ 5892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.358432 (+ 6622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.362260 (+ 3828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.363019 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.363137 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.366263 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:33.366271 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:33.366279 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:33.366323 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:33.370194 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.370976 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.370982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.370985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.371144 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.371751 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.371760 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.371766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.372173 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:33.372449 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:33.373316 (+ 867us) server_negotiation.cc:299] Negotiation successful
0925 01:22:33.373567 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":62}
I20260925 01:22:33.374799 11436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.338201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41512 (local address 127.20.102.188:36559)
0925 01:22:33.351075 (+ 12874us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.351084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.351109 (+ 25us) server_negotiation.cc:407] Connection header received
0925 01:22:33.351163 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.351168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.351255 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.352482 (+ 1227us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.357015 (+ 4533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.359437 (+ 2422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.361241 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.361871 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.362000 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.365886 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:33.365892 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:33.365897 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:33.365938 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:33.370365 (+ 4427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.371137 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.371141 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.371144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.371322 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.371776 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.371782 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.371785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.372216 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:33.372347 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:33.373941 (+ 1594us) server_negotiation.cc:299] Negotiation successful
0925 01:22:33.374369 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":12712,"thread_start_us":13139,"threads_started":1}
W20260925 01:22:33.376962 11346 tablet.cc:2369] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 01:22:33.378161 11367 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075 [term 1 FOLLOWER]: Refusing update from remote peer 0776c251dffa401b9e3d74083e3d53e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:22:33.379154 11413 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:22:33.380070 11396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:33.327783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.20.102.190:33417)
0925 01:22:33.327988 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.327997 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.335826 (+ 7829us) server_negotiation.cc:407] Connection header received
0925 01:22:33.336701 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.336707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.336791 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.336866 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.348731 (+ 11865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.349881 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.352992 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:33.353712 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:33.353838 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:33.365956 (+ 12118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 01:22:33.365965 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 01:22:33.365972 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 01:22:33.366016 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 01:22:33.376277 (+ 10261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.377223 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.377229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.377232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.377383 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 01:22:33.377940 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 01:22:33.377946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 01:22:33.377949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 01:22:33.378441 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 01:22:33.378728 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:33.379335 (+ 607us) server_negotiation.cc:299] Negotiation successful
0925 01:22:33.379703 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 01:22:33.385490 11318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86: Bootstrap starting.
I20260925 01:22:33.390726 11318 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86: Neither blocks nor log segments found. Creating new log.
I20260925 01:22:33.392817 11318 log.cc:824] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86: Log is configured to *not* fsync() on all Append() calls
I20260925 01:22:33.401649 11318 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86: No bootstrap required, opened a new log
I20260925 01:22:33.401625 11399 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0776c251dffa401b9e3d74083e3d53e2. Latest consensus state: current_term: 1 leader_uuid: "0776c251dffa401b9e3d74083e3d53e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } } }
I20260925 01:22:33.401952 11399 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:22:33.405953 11437 mvcc.cc:205] Tried to move back new op lower bound from 7333066151425019904 to 7333066151178018816. Current Snapshot: MvccSnapshot[applied={T|T < 7333066151425019904}]
I20260925 01:22:33.407702 11438 mvcc.cc:205] Tried to move back new op lower bound from 7333066151425019904 to 7333066151178018816. Current Snapshot: MvccSnapshot[applied={T|T < 7333066151425019904}]
I20260925 01:22:33.416193 11318 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } }
I20260925 01:22:33.416704 11318 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b90e6daf4b64d17ab1354c7297bae86, State: Initialized, Role: FOLLOWER
I20260925 01:22:33.417320 11318 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } }
I20260925 01:22:33.424611 11443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } } }
I20260925 01:22:33.427320 11293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0776c251dffa401b9e3d74083e3d53e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9b90e6daf4b64d17ab1354c7297bae86"
I20260925 01:22:33.427790 11292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86 [term 0 FOLLOWER]: Advancing to term 1
I20260925 01:22:33.427290 11318 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 01:22:33.429888 11443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:22:33.431843 11292 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86 [term 1 FOLLOWER]: Refusing update from remote peer 0776c251dffa401b9e3d74083e3d53e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 01:22:33.432933 11427 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 01:22:33.434789 11427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0776c251dffa401b9e3d74083e3d53e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } } }
I20260925 01:22:33.435058 11427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:33.435137 11293 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0776c251dffa401b9e3d74083e3d53e2 in term 1.
I20260925 01:22:33.438117 11434 catalog_manager.cc:1347] Generated new cluster ID: 5ab6f9c018794a9483d0528183a2d038
I20260925 01:22:33.438222 11434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 01:22:33.439383 11399 sys_catalog.cc:455] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0776c251dffa401b9e3d74083e3d53e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } } }
I20260925 01:22:33.439632 11399 sys_catalog.cc:458] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:22:33.469388 11443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0776c251dffa401b9e3d74083e3d53e2. Latest consensus state: current_term: 1 leader_uuid: "0776c251dffa401b9e3d74083e3d53e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } } }
I20260925 01:22:33.469715 11443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:22:33.470176 11448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0776c251dffa401b9e3d74083e3d53e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } } }
I20260925 01:22:33.470515 11448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 01:22:33.509194 11413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0776c251dffa401b9e3d74083e3d53e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0776c251dffa401b9e3d74083e3d53e2" member_type: VOTER last_known_addr { host: "127.20.102.190" port: 33417 } } peers { permanent_uuid: "9b90e6daf4b64d17ab1354c7297bae86" member_type: VOTER last_known_addr { host: "127.20.102.189" port: 42977 } } peers { permanent_uuid: "e65c16a64b2d49039193a4ef47c2d075" member_type: VOTER last_known_addr { host: "127.20.102.188" port: 36559 } } }
I20260925 01:22:33.509557 11413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2 [sys.catalog]: This master's current role is: LEADER
I20260925 01:22:33.513590 11434 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 01:22:33.522646 11434 catalog_manager.cc:1504] Loading token signing keys...
I20260925 01:22:33.544888 11458 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 01:22:33.557442 11434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0776c251dffa401b9e3d74083e3d53e2: Generated new TSK 0
I20260925 01:22:33.558580 11434 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 01:22:33.665808 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:33.649740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.20.102.189:42977)
0925 01:22:33.650344 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.650352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.661252 (+ 10900us) server_negotiation.cc:407] Connection header received
0925 01:22:33.661604 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.661613 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.661729 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.661825 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.665360 (+ 3535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45564: BlockingRecv error: recv got EOF from 127.0.0.1:45564 (error 108)
Metrics: {"server-negotiator.queue_time_us":430}
W20260925 01:22:33.666807 11396 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:33.650532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38020 (local address 127.20.102.190:33417)
0925 01:22:33.650799 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.650807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.661254 (+ 10447us) server_negotiation.cc:407] Connection header received
0925 01:22:33.662069 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.662075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.662172 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.662264 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.666577 (+ 4313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38020: BlockingRecv error: recv got EOF from 127.0.0.1:38020 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 01:22:33.669436 11436 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:33.661454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.20.102.188:36559)
0925 01:22:33.661706 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.661713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.667432 (+ 5719us) server_negotiation.cc:407] Connection header received
0925 01:22:33.668083 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.668089 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.668185 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.668255 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.669215 (+ 960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41514: BlockingRecv error: recv got EOF from 127.0.0.1:41514 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 01:22:33.679572 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:33.677306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.20.102.189:42977)
0925 01:22:33.677473 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.677478 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.678056 (+ 578us) server_negotiation.cc:407] Connection header received
0925 01:22:33.678284 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.678289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.678432 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.678548 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.679336 (+ 788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45566: BlockingRecv error: recv got EOF from 127.0.0.1:45566 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260925 01:22:33.688954 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:33.686717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45572 (local address 127.20.102.189:42977)
0925 01:22:33.686896 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.686903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.687454 (+ 551us) server_negotiation.cc:407] Connection header received
0925 01:22:33.687626 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.687631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.687717 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.687820 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.688754 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45572: BlockingRecv error: recv got EOF from 127.0.0.1:45572 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 01:22:33.696125 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:33.694062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.20.102.189:42977)
0925 01:22:33.694246 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.694252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.694745 (+ 493us) server_negotiation.cc:407] Connection header received
0925 01:22:33.694923 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.694929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.695017 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.695113 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.695882 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45584: BlockingRecv error: recv got EOF from 127.0.0.1:45584 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 01:22:33.708438 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:33.706297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45594 (local address 127.20.102.189:42977)
0925 01:22:33.706463 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.706469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.706995 (+ 526us) server_negotiation.cc:407] Connection header received
0925 01:22:33.707188 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.707194 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.707293 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.707405 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.708231 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45594: BlockingRecv error: recv got EOF from 127.0.0.1:45594 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 01:22:33.721864 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:33.719659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.20.102.189:42977)
0925 01:22:33.719832 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.719839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.720381 (+ 542us) server_negotiation.cc:407] Connection header received
0925 01:22:33.720569 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.720576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.720671 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.720801 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.721633 (+ 832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45608: BlockingRecv error: recv got EOF from 127.0.0.1:45608 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 01:22:33.744041 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:33.741685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.20.102.189:42977)
0925 01:22:33.741882 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.741889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.742448 (+ 559us) server_negotiation.cc:407] Connection header received
0925 01:22:33.742663 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.742668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.742770 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.742892 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.743803 (+ 911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45624: BlockingRecv error: recv got EOF from 127.0.0.1:45624 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260925 01:22:33.789045 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:33.786429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.20.102.189:42977)
0925 01:22:33.786625 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.786633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.787206 (+ 573us) server_negotiation.cc:407] Connection header received
0925 01:22:33.787447 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.787454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.787570 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.787702 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.788757 (+ 1055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45638: BlockingRecv error: recv got EOF from 127.0.0.1:45638 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 01:22:33.859643 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:33.857476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.20.102.189:42977)
0925 01:22:33.857722 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.857728 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.858066 (+ 338us) server_negotiation.cc:407] Connection header received
0925 01:22:33.858279 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.858283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.858362 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.858453 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.859405 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45642: BlockingRecv error: recv got EOF from 127.0.0.1:45642 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
W20260925 01:22:33.993798 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:33.991656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.20.102.189:42977)
0925 01:22:33.991906 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:33.991913 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:33.992235 (+ 322us) server_negotiation.cc:407] Connection header received
0925 01:22:33.992406 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:33.992411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:33.992484 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:33.992613 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:33.993504 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45654: BlockingRecv error: recv got EOF from 127.0.0.1:45654 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 01:22:34.252415 11410 catalog_manager.cc:1259] Loaded cluster ID: 5ab6f9c018794a9483d0528183a2d038
I20260925 01:22:34.252560 11410 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075: loading cluster ID for follower catalog manager: success
I20260925 01:22:34.255685 11410 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075: acquiring CA information for follower catalog manager: success
W20260925 01:22:34.256568 11322 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 01:22:34.254532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45670 (local address 127.20.102.189:42977)
0925 01:22:34.254695 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:34.254699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:34.255076 (+ 377us) server_negotiation.cc:407] Connection header received
0925 01:22:34.255280 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:34.255286 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:34.255382 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:34.255510 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 01:22:34.256337 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45670: BlockingRecv error: recv got EOF from 127.0.0.1:45670 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 01:22:34.257994 11410 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e65c16a64b2d49039193a4ef47c2d075: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:22:34.549286 11458 catalog_manager.cc:1259] Loaded cluster ID: 5ab6f9c018794a9483d0528183a2d038
I20260925 01:22:34.549394 11458 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86: loading cluster ID for follower catalog manager: success
I20260925 01:22:34.551993 11458 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86: acquiring CA information for follower catalog manager: success
I20260925 01:22:34.553839 11458 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b90e6daf4b64d17ab1354c7297bae86: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 01:22:34.783365 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:34.774406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45680 (local address 127.20.102.189:42977)
0925 01:22:34.777547 (+ 3141us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:34.777552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:34.777568 (+ 16us) server_negotiation.cc:407] Connection header received
0925 01:22:34.777637 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:34.777640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:34.777866 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:34.778017 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:22:34.778474 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:34.779410 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:34.780764 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:34.781249 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:34.781556 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:34.782084 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:22:34.782675 (+ 591us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:22:34.782855 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:34.782911 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 01:22:34.783000 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2973,"thread_start_us":2810,"threads_started":1}
I20260925 01:22:34.795629 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:34.789525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.20.102.188:36559)
0925 01:22:34.790181 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:34.790189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:34.790209 (+ 20us) server_negotiation.cc:407] Connection header received
0925 01:22:34.790330 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:34.790334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:34.790486 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:34.790668 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:22:34.791175 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:34.791898 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:34.793335 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:34.793749 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:34.793887 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:34.794396 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:22:34.794934 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:22:34.795091 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:34.795218 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 01:22:34.795323 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":411,"threads_started":1}
I20260925 01:22:34.815032 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:34.802287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.20.102.189:42977)
0925 01:22:34.802474 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:34.802480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:34.805900 (+ 3420us) server_negotiation.cc:407] Connection header received
0925 01:22:34.807268 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:34.807277 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:34.807404 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:34.807482 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:22:34.808406 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:34.809977 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:34.812671 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:34.813395 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:34.813491 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:34.814190 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:22:34.814318 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:22:34.814381 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:34.814623 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 01:22:34.814706 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 01:22:34.820171 11536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:34.801818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38022 (local address 127.20.102.190:33417)
0925 01:22:34.812579 (+ 10761us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:34.812588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:34.812609 (+ 21us) server_negotiation.cc:407] Connection header received
0925 01:22:34.812844 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:34.812850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:34.813079 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:34.813883 (+ 804us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:22:34.814086 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:34.815193 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:34.817334 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:34.817942 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:34.818074 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:34.818815 (+ 741us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:22:34.819256 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:22:34.819766 (+ 510us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:34.819827 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 01:22:34.819919 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10625,"thread_start_us":10856,"threads_started":1}
I20260925 01:22:34.822057 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 01:22:34.805805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41526 (local address 127.20.102.188:36559)
0925 01:22:34.806015 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 01:22:34.806023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 01:22:34.815793 (+ 9770us) server_negotiation.cc:407] Connection header received
0925 01:22:34.816065 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 01:22:34.816071 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 01:22:34.816162 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 01:22:34.816223 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 01:22:34.816809 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:34.817901 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:34.819959 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 01:22:34.820625 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 01:22:34.820800 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 01:22:34.821270 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 01:22:34.821416 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 01:22:34.821520 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 01:22:34.821692 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 01:22:34.821785 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 01:22:34.825162 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 11177
I20260925 01:22:34.826296 11239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:22:34.937942 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 11177
I20260925 01:22:34.963127 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 11251
I20260925 01:22:34.964362 11313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:22:35.073136 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 11251
I20260925 01:22:35.093762 20890 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task_TC7zy/build/asan/bin/kudu pid 11325
I20260925 01:22:35.094959 11387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 01:22:35.206897 20890 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task_TC7zy/build/asan/bin/kudu with pid 11325
2026-09-25T01:22:35Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4917 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9967 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (365176 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST